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© A parcel array processor tor massively parallel applications is formed k>v* pr>ver CMOS with DRAM 
processing while incorporating prooeseing stements on a singte chip- Eight processor* on e angle cfiip have 

a their own associated processing element, significant memory, and I/O and ere interconnected with a hypetcube 
based, but modified topotogy. These nodes are then Interconnected, efcher by a hypercuoe, modified hyper- 
01 cube, or ring, or ring within ring network topology. Conventional microprocessor MMPs consume pins and time 
Cs| going to memory. The new arenrtocsure merges processor and memory with multiple PMEs (eight i$ Wt 
processors v«th 32K and I/O) in DRAM and has no memory access delays and uses ail the pins tor nefroridftg. 
q The chip can be a single node of a fine-grained parallel processor Each chap wiS have eight 16 bit processors, 
IN each processor proving 5 mips performance- I/O has three Internal porta and one edernal port shored by the 
U> plural processors on ttte chip. Significant scrftware ttoribftty is provided to enatte quick impJementation of 
O existing programs written In common languages. II fc a developable and expandable technology without need to 

& develop new pinouto. new software, or new utilities eo chip density inw&eaee ond new hardware b provided for e 
chip function. The scalable chip PME hea internoJ end external connectiona tor broadcast find asynchronous 
SIMD, fellMD and SlMIMD (SlMD/MIMD) nim dynamic awfechtng of modes. "Hie cfcip can be used In systems 
v?hich employ 32. 64 or 128,000 processors, and can be used for lower, intermediate and higher rai»g*e. Local 
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and global memory functions can all be provided by (lie chips themselves, and Die system can connect to and 
support (Slier globe) memories and DASO. The chip can be used as a microprocessor accelerator, In personal 
computer applications, ae a vision or avionics computer system, or as workstation or supercomputer. There ie 
program compaabiWy lor the fWly scalable system. 
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FIELD OF THE INDENTIONS 

The invention relate* to compute aw* computer system* and particularly to parallel array processor*. 
In accordance wrih the invention, a parallel array processor tfPAP) may be incorporated on a single 
semiconductor silicon chip. This chip forms a basis for the systems described which are capable o1 
massively parallel processing erf complex scteitflte and business appiteetlons, 

REFERENCES USED IN THE DISCUSSION OF THE INVENTIONS 

In the detailed discussion of the invention, other works vrtU be referenced, including references to our 
own unpublished works which are not Prior Art which will aid the reader fcn following the discussion. 

GLOSSARY OF TERMS 

o ALU 

ALU Is the arithmetic logic unit portion or a processor, 
0 Array 

Array refers to an arrangemeni oi elements in one or more dimensions. An array can include an 
ordered sat of daze Hems {array element) which in languages Ifte Fortran are Identified by a single 
name, in other languages such a name of an ordered set of data items refers to an ordered coflecfion 
or set of data elements, afl of which have identical attributes. A program array has dimen s brc 
specified, generally by a number or dimension attribute. The declarator of Use array may also specify 
^e size of each dimension of tie array in soma languages, in some languages, an array is an 
arrangement of elements in a table. In a hardware sense, an array is a collection of structures 
(functional elements] wWch are generally Identical in a massively paraBel archrtecBure. Array elements 
in data parallel computing are elements which car* be assigned operations and when parallel can each 
independency and in parallel execute the operations required. Generally, arrays may be thought of as 
grids erf piocessing elements, Secflons of the array may be assigned sectional data, so that sectional 
data can be moved around in a regular qn& pattern. However, dam can be indexed or assigned to an 
arbitrary location in an array. 
0 Array Director 

An Array Director is a unit prograrrwned as a control fer fur an array- It performs the function of a 
master controller for a grouping of Functional elements arranged in an array, 
o Array Processor 

There two principal types ot array processors • multiple instruction rnuBlpte date (MIMD) and stngts 
Instruction multiple dale (SIMD). in * mud array processor, each processing element m the array 
executes its own unique instruction stream wifri its own data, b a SWD array processor, each 
processing element in the array is resiricled to the same instruction via a common instruction stream; 
however, the data associated \rfth each processing element Is unique. Our preferred array processor 
has other cftaractensfc*. We call * Advanced Parallel Array Processor, and use the acronym APAP, 

o Asynchronous 

Asynchronous is without a regular time relaflorehip; the e*ecutioo of a function is unpredictable with 
respect to me execution ot other Junctions which occur wlmout a regular or predictable time 
ralatfansrap to other (unction exacutSona In control situations, a controller will address a location to 
which control is passed when data is waiting ior an idle element being addressed This permits 
operations to remain in a sequence while tfiey are out of time coincidence with any event 
o BOP&GOP6 

BOPS or OOPS ere acronyms having the same meaning - b2Kon$ of operations per second. See 

aops. 

o Circuit Switched/Store Forward 

These terms refer to two mecfianisms for moving data packets through a network ot node*. Store 
Forward is a mechanism whereby a data packet Is received by each intermediate node, stored into its 
memory, and Ihen forwarded on towards Its destination. Circuit Switch Is a mechanism whereby an 
intennetfate node is commanded to wgicaiiy conned its Input port io an oujhit port such that da*a 
pockets can pass dirccfly through the node towards ttsir destination, without entering the intermediate 
node's memory. 

o Cluster 

A cluster ia a Stefan <or functional unit) which consists of a control unit (cluster controller) and the 
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hardware (which may be terminals, functional units, or virtual components) stiached to it. Our Cluster 
Includes an array of PMEs sometimes c^ted a Node array. Usually a cluster has 612 PMEs. 

Our Entire PME node array consists of a set of dusters, each caster Supported by a dusfei 
corrtrdler (CC), 
o Cluster controller 

A duster oexnrtfief is a device that controls fnputfoutput <l/0) operations for more tnan one device oi 
fonctional unit connected to it A duster corrorotier is usually comroJIed by a program stored and 
executed In the unK as ft was In the IBM 3601 Finance Oomraunlcaiion Controller, but It can be 
entirely controlled by hardware as it tvae in ihe IBM 3272 Control Unit 
o Cluster synchronizer 

a cJuster synchronizer to a functional una which manages the operations of ell or pert of a duster to 
maintain synchronous operation of the elements so iflat the functional units maintain a particular time 
relationship with the execution of a program, 
o Controller 

A con to Iter is a device that directs the transmission of data and instructions over the links of an 
interconnection network: Its operation Is controlled by a program executed by a processor to which 
the controller is connected or by a program executed within the device, 
o CMOS 

CMOS Is an acronym for Complementary Metal-Oxide Semiconductor technology. It is commonly 
used to manufacture dynamic random access memories <DRAM$X NMOS is another technology used 
to manufacture DRAMS, We prefer CMOS but the technology used to manufacture the APAP is nm 
intended to fimH the scope of the semiconductor isehnofcgy *Mcn Is employed, 
o Dotting 

Dotting refers to the jtintng of throe or more leads by physically connecting them together. Most 
backpane* busses share this connection approach. The term relates to Oft DOTS of times past but Is 
used here to identify multiple data sources that can be combined onto a bus by a very simple 
protocol. 

Our I/O zipper concept can t» used to Implement concept that the port into e node could be 
driven by the port out o? a node or by data coming from the system but Conversely, data being put 
out of a node *>oukJ be avaiable to boft too input to another node and to the system bus. Note that 
outpuifing data to both tte system bus and another node is not done simylfoneously bus in different 
cycles. 

Dotting Is used In fre H-DOT discussions where Tvvo-porbsd PEs or PMEs or Pickets can be used 
in array* o? various organizations by taking advantage of dotting. Several topobgtes are discussed 
incfcjcftng 20 and 3D Moshes. Base 2 N-cube. Sparse Bass 4 N-cubo, and Spare© Base 8 N-cube. 
0 DRAM 

DRAM is an acronym tor dynamic random access memory, the common storage used by computers 
for main memory. However, the term DRAM can be applied to use as a cache or as a memory which 
Is not the main memory. 
0 FLOATING-POINT 

A floating-poinl number is expressed in two parts- There is a fixed point or fraction pari, end an 
exponent part to some assumed radix or Base. The exponent indicates the actual placement of the 
decimal point. In the typical floaang-poim representation a real number 0,0001 23* is represented as 
0.1234-3. where 0.1234 is the lixod-poaii part and O is She exponent. In this example, lha floating- 
point radix or base is 10, where 10 represents lb© implicit fixed poeWve integer base* greater Wan 
unity, that is raised to the power explicitly denoted by the exponent in the floating-point representation 
or represented by the characteristic in the floating- point representation and nan multipfied by (he 
fixeoVpofrit part to determine the real number represented. Numeric literals can be expressed in 
floatlng-pctm notation as wea as real numbers, 
o FLOPS 

This terms refers to flcaiinprtofri instructions per second. Floating-point operations include ADD. 
SUB, MPY, OiV and often many others. Floating-point Instructions per second parameter is often 
calculated using the add or multiply instructions and. in general* may be considered to have a 50/50 
mix. An operation includes the generation ot exponent, fraction and any required fraction normaliza- 
tion. We could aoV**se 32 or 40-bit floating-point formats for longer but vre have not counted them In 
m mix.} A floating-point operation when implemented with fixed point Instructions (normal or" RISO 
retires multiple instructions. Some use a 10 to 1 ratio in figuring performance while come specific 
studies have shown a ratio of 6.25 more appropriate to use, Various architectures will have different 
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ratios, 
o functional unit 

A functional unit is an enWy of hardware*, software, or both, capable of accomplishing a purpose, 
o Sbytes 

5 Gbytes refers to a bison bytes. Gbyte&'s would be a biRlon bytes per second, 

o GIGAFLOPS 

Cl0r9 floating-point Insaudtons pe* second- 
o QOPSand PETAQP9 

QOP8 or BOP$> have the same meaning - buttons of operations per second PETAOPS means 
jo trillions of operations per ascend, a potential of the current machine. For our APAP rnachine t>ey are 

jusi about the same as BIPs-QFs meaning batons of instructions per sscond. tn some machines an 
instruction can cause too or more operations (le. both an add find multiply) but *"e don't do that. 
Ahernetively It could tate many irotfticiion* to do an op. For exarapto we use multiple infraction* to 
perform 64 bit arithmetic- In counting ops however, we did not elect to count log ops, QOPS may bo 
» !he preferred use to describe performance, but there Is no consistency In usage that has been noted. 
One seas MIPa-MOPs then 8IF*/BOPs and Megaf LOPSr'GioaFLOPS'T«raFLOPSrP^taFiopd. 
o ISA 

ISA meena the Instruction Set Architecture, 
o Link 

so A link is an element which may bo physical or logical. A physical link Is the physical connection for 

Joining elements oi unris* while in computer progiamtnlng a link la an instruction or eddiess that 
passes control and parsrnettins between separate portions of the program, m multisystems a link te 
ihe connection between fevo systems which may b® cpccifled by program code Identifying thd link 
which may be identified by a real or virtual address. Thus generally e Jink includes ihe physical 

cs medium, any protocol, and associated devices and programming; it is both logical and physical. 

o m flops 

MFLOPS means (tore floating-point inductions per second, 
o MIMD 

MlfclD is used to reter to a processor array archtistiure wherein each processor in the array has its 
30 own instruction stream, thus Multiple Instruction stream, to execute Muhiple Data streams located one 
per processing element, 
o Module 

A module is a program una that & discrete and identifiable or a functional unit of hardware designed 
for use vrtth other components. Also, a collection of PGs contained in a single electronic chip is called 
» a module- 
o Mods 

Generally, a node is ffte junction or links, in a generic array oi PEb h one PE can be a node. A node 
can also contain a collection of pes called a module, to accordance wm\ out invention a node is 
formed of an array of PMEs, and we refer to the set of PMEs as a node. Preferably a node is 8 PMEs, 
<o a Node army 

A collection of modules mads up of PMEs is sometimes referred to as a node array, is an array of 
nodes made up of modules- A node aray is usually more *an a few PMEs, but the term 
encompasses a plurality, 
o POE 

<s A PDE is a partial differentia} equation, 

o PDE relaxation solution process 

PDE relaxation solution process is a way to solve a PC€ (partial differentia] equation}. Solving PDEe 
uses most of the super computing compute power in the known universe and can therefore be a good 
example of relaxation process- Tnere are m*ny ways to solve the pde equation and more than 

so one of the numerical mefrrods includes the relaxation process. For example, if a PDE is solved by 
finite element rnattods relaxation consumes the bulk of the computing lima. Consider an example 
from the world of heat transfer. Given hot gas Inside a chimney and a oold wind outside, how wffl the 
temperature gradient within ** chimney bricks develop? By consJofertng She bricks as tiny segments 
and writing an aquation mat says how heal flows Between segments as a lunctlon of toitperatura 

ss differences then the heat transfer PDE has been converted «nfo a finite element problem. If we then 
say all elements except those on the inside and outside are at room tempereSure while the boundary 
segments are at (he hot gas and cold vrind temperature, we have set up the problem to begin 
relaxation. The computer program then modera time by updating the temperature variable tn each 
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segment based upon the amount of Iras thai flows into or out of the segment It takes many cycfes of 
proofing an [he segments In the model before fte set of temperature variables across the chimney 
relaxes \o repressnt actual temperature distrfcutkm that would oca* in th& physical chimney. H tl»e 
obfccflve was to model 90$ cooling in me chimney thwi Bie ©toments wild haw to extend to gas 

s equations, and the boundary conditions on the inside would be linked to another finite element model, 

and fha process oon&ftu»c- Note that the hee* flow is dependant upon the temperature difference 
between the eagmern and Re neighbors, k thus uses the inter-PE communication paths to dsftibute 
the temperature variables. R i* this near neighbor communcatlon pattern or characteristic that makes 
PDE reJaflon very applicable to parallel computing. 

to 0 PICKET 

This <s the element in an ariay 01 elements making up an array processor, it consists ofc data flow 
<ALU REGS), memory, control, and the portion oi the cofftnwfcaiion matrix essocraied with the 
element. The unit refers to a 1/nth ot an array processor made up of parallei processor and memory 
efemants w«h *ieir control and portion of the array Intercommunion mechanism. A picket is a form 

*5 of processor mernory element or PWIE-. Our PME chip design processor logic can irnpiernent the 

picket logic described in related applications or have the logic for the array of processors formed as a 
node. The term PICKET is simile* to the commonly used array term PE for processing element* and 
fa an element of the processing array preferably comprised of a combined processing ©foment end 
toed memory for processing bit parallel bytes of information In a clock cycle. The preferred 

20 embo&mem consisgnrj of a byte wfede data flow proceseor. 32k byies or more of memory, primitive 
controls and ties to communications witti other pickers. 

The term "picker comes from Tom Sawyer and his white fence, although it will a?so be 
understood functionally that a military picket line analogy fne quit* **|| r 
© Picket Chip 

23 A picket chip contains a plurality of pickets on a single silicon chip, 

0 Picket Processor system (or Subsystem) 

A picket processor is a lata* system consisting or an array of picket*, a ccmrminicalion network, an 

VO system, and a SIUD controller consisting of a microprocessor, a canned routine processor, and a 

microcontroller that runs the array. 
30 o Picket Architecture 

The Picket Architecture is the preferred embodiment for tl>e SIMD architecture writs features thai 

actxrmmodsie several diverse kinds of problems including: 

- set associative processing 

- parallel rnimericaJJy intensive processing 

33 * physical array processing similar to images 

0 Picket Array 

A packet array is a collection of pic*Ms anai>gecl in a geometric order, a regular array, 
0 PME or processor memory element 

PME is used to a processor merncry element We use the term PME to refer * a stogie processor. 

* memory and W) capable system element or unit that forms one of our parallel array processors. A 
processor memory element is a term which encompasses a picket A processor memory element is 
tmtb of a processor array tvtiich comprises a processor* tts associated memory, control Interface, and 
a person ot an array communication networtc mechanism. This element can have a processor memory 
element with a connectivity ot a regular array, as m a picket processor, or as part of a subarray, as in 

<s the multi-processor memory earner* node ^e have described- 
o Routing 

Routing is the assignment of a physical path by which a message will reach its destination. Routing 
assignments have a source or origin and a destination. These elements or addresses have a 
temporary relationship or affinity. Often, message routing is based upon a key which is obtained by 
w reference to a table of assignments. In a network, a destination is any station or network addressable 

unrt addressed as the desnnahon ot information transmitted by a path control address that identifies 
the Knk, The destination field identifies the destination with a message header destination code, 
0 $IMD 

A processor array archaecture wherein ail processors in the array are commanded from a Single 
as Instruction stream to execute Mulfipk Data streams located one per processing ©lemenL 

0 $MDMIMD or &MDMIMD 

SWDMiMD or StMD/MIMD is a farm referring to a machine iharf has a dual fundton ttat can switch 
from MIMD to SttvID tor a period of time to handle some complex Instruction, ana Slus hss tno 
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modes. The Thinking Machines, he. Connection Machine model CM-£ when placed &B a buiii end or 
back end of a MIMD machine permitted programmers to operate different modes for execution of 
different parte of a probfem. rewerrod to comet^ec $ dual mode* Thee* machine* have existed since 
Mac and have smployscl a bus that interconnects the master CPU with airier processors. The master 
9 control processor would have the capability of Interrupting the processing of other CPUs. The other 

CPUs could run independent program code. During an interruption, some provision must be made for 
checkpointing (dosing and saving current status of the conrrolisd processors). 
0 BIMIMD 

SJM1MD id e processor array architecture wherein ail proceeds in the army are commanded from a 
?o Single Instruction stream, to execute Multiple Data streams located one per processing element. 

Wiffiln this construct, date dependent operations within each picket that mimic instruction execution 

are control ted by the a WO instruction stream. 

This is a &ngte Instruction Stream machine with fte abiay to sequence Multiple foEtrueiion 

streams (one per Picket) using the SlMD instruction stream and operate on Multiple Data Streams 
;s (one per Picket). SIMIMD con be executed by e processor memory etarrrcrri system. 

Sf$D 

SI8D ss an acronym for Single Instruction Single Data, 
*0 o Swapping 

Swapping interchanges the data content of a storage area with that of another area of storage. 

o Synchronous Operation 

Synchronous operation in a MIMD machine is a mode of operation In which each action i$ related to 
an event jusuafly a clock); il can be a specified event that occurs regularly in a program sequence. An 
operation Is dispatched to a number of PEs who then go off to Independently perform Pie fonctlon. 
Control is not retimed to the controller unta the operafion k cornpisied. If the request te to an array of 
bnedonsl unite, the request is generated b? a controller to elemente In toe array wWch roust complete 
their operation before control Is returned to the controller. 

0 TBRAFLOPS 

TERAFLOPS means (lor 12 ftoatfng-poN^ Instructions per second, 
o VLSI 

VLSI is an acronym for very large scale integration (as applied to integrated circuits}, 
o Zipper 

A zipper is a new fcjnctlon provided. It allows for links to be made from devices which are external to 
35 ir» normal inrerconnsctkxj of an array configuration. 

BACKGROUND OF THE INVENTION 

In the never ending quest lor faster computers, engineers are linking hundreds, end even thousands of 
4fl low cost mtaooroceaeore together in parallel to create super super computer* that divide in order to conquer 
complex problems that stump today's machine*. Such machtnas are called massively parade). We have 
created a new way to create massively parallel systems. The many improvements *hteh we have made 
should be considered against the background of marry writs of others. 

Multiple computers operating In parallel have existed for decades. Early parallel machines included lha 
« iluac which vuee started in tne 1980s, iluac rv was butt in the 19709. other multiple processors Include 
(see a partial summary m U.S. Patent 4#75,8$4 Issued Decembers ifign to Xu et al) the Cedar, S>gm«H, 
the Butterfly and the Monarch, the Wei fcac. The Connection Machines, the Cattech COSMIC, (he N Cube, 
IBM's RP& IBM's 6F11 , ihe NYU Ultra Computer, the Intel Defte and Toucfeaone. 

Large multiple processors beginning with ILUAC have been considered stipercomputere. Supsrcom- 
so puters wiili greatest commercial success have been based upon multiple rector processors, represented by 
the Cray Research Y-MP systems, me IBM 3090, and other manufacturer* machines mciuding those of 
Amdahl, Hftachi, Fujitsu, and NEC 

Massively PereSeJ Processors (MPPs) are now thought of as capable of beoomtog supercomputers. 
These computer systems aggregate a large number ot rr^cro processors wftn an Interconnecsicn network 
ss and program ffiem to operate in parallel. There have been hvo modes of operation of these computers. 
Some of these machines have been MIMD mode machines. 

Some of these machines have been SlMD mode machines. Perhaps th© most commercially acclaimed 
of these machines has been the Connection Machines series 1 and 2 of Thinking Machines. Inc , These 
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have been esssfraalry SIMD machines. Many of the massively paraltei machines havs used microprocessors 
Interconnected In parallel to obtain (heir concurrency or parallel operations capablSty. Intel rnteroprooe sears 
Kfee i860 neve been used by mtei other©. N Cube has made euch m$o«fi« ** Intel r 389 
rotoroprocessore. Other machines have been built with what is called the "transputer" chip, tamos 

5 Transputer IMS T800 Is an example. The feimos Transputer T800 Is a 32 bit device wllh an Integral high 
speed floating point processor 

Ae an example of tto kind of systems that are bm t several Irirnos Transputer T800 chips each would 
neve 32 communication Unit Inputs and 32 link outputs. Each cWp vwld have a single processor, a small 
amount of memory, end communication links to the focal memory and to an external interfacs- In addition, 

jo In order to buBd up Ste system communication Snk adaptors fike IMS C011 and 0012 would be connected, 
in addition switches, luce e IMS 0004 would provide, say, a crossbar swr&cn between the 32 ttr* inputs and 
32 link outputs to provide pdrt-to-poirti connection between additional transputer chips. In addHfon, there 
wvai be special oiroufry and interface chips for transputers adapting them to be ueed For a special purpose 
tailored to the requirements of a specie device, a Graphics or disk controller . The tamos IMS M212 is a 10 

j s bit processor, wtti on cWp memory and oommunlcstion finks. It contains hardware and logic to control cfisk 
dives and can be used as a programmable disk controller or as a general purpose interlace. In order lo use 
the concurrency (parallel operations} Inmos developed a special language, Occam, for the transputer. 
Programmers have to describe the network of transputers dorecftiy in an Occam program. 

Some or these massively parallel machines use paraSet processor arrays of processor chips tthlch are 

20 inierconnected mf> different ropoJorji&s. The transputer provides a crossbar network wrii the addition of IMS 
C004 chips- Sonne other systems use a hypercube connecsort- Others use a bus or mesh to connect the 
microprocessors end there associated circuitry. Some have been Interconnected by ctreurt switch proces- 
sors that use sv£ch*& as processor addressable networks. Generally, as wfth ft* 14 RtSCtiQOOs which 
were interconnected last (all at Lawrence Uvermore by wiring the machines together, the processor 

as addressable networks have been considered as ooatse-grained multiprocessors* 

Some very large machine sre being built by Intel and nCube and otters «> sitae* what ere called 
"grand chaUsnges" in data processing. However, these computers are very expensive, Recent projected 
costs ero In the order of $30,000,00000 to $75,000,000-00 (Tera Computer) for computers whose 
envelopment has astn funded by me US Government to attack the "grand challenges*. These ^rand 

ao challenges" would Include such problems ae oSiroats modeling, fluid turbulence, pollution dispersion, 
mapping of the human genome aiKl ocean c»c^eSon t quantum cferomodynamics, semiconductor and 
supercomputer modeling, combustion systems, vision and cognSoa 

As a footnote to our bacftground, we should recognize one of the early massively parallel machines 
developed by IBM. Id our description we have chosen to use the term processor memory element rather 

as than "tranepuier" to describe one of the sight or more memory units wiJh processor and I/O capabtlroes 
whfch make up the array of PMEs io a chrjx or nods. The referenced prior art "transputer" has on a chip 
one processor, & Fortran coprocessor, find a smalt memory, wnh an I/O irrierface. Our processor memory 
element could apply to a transputer and to the PME of the RP3 generaSy. However, as will be recognized, 
our Bttte chip is significantly dKTeren* In many respects. Our little chip has many features described later. 

40 However, we do recognise that t*e term PME was tiret coined tot another, now more typical. PME vmieh 
formed the basis for the massively parallel machine known as the RP& The IBM Research Parallel 
Processing Prototype (RP3) was an experimental parallel processor based on a Multiple instruction Multiple 
Data (MIMD) archseoure. RP3 was designed and bum 82 IBM TJ. Wafaon Research Center in cooperation 
with lha New York University Uttracomputar project. This work was sponsored In part by Defense Advanced 

*s Research Project Agency . was comprised of 94 Prcoessor-Memory Elements (PMEe) interconnected 
by a hrgh speed omega network Each PME contained a 32-bit IBM *PC scientific" microprocessor, 3£-k8 
cache, a <VMB segment of me system memory „ and an 1*0 port. The PME I/O port hardware end eoltvvare 
supported ieltiarfzagon, status acquisition, as well as memory and processor communication through shared 
I/O support Processors (ISPs). Each I8P supports oicht processor- memory elements through the Extended 

so \fO adapters (ETlOs), independent of (he system network. Each ISP Interfaced to the IBM 3/370 channel 
and the IBM Toker>Ring network as well as providing operator mentor service. Each extended vo ac&pfor 
attached os a device to a PME ROMP Storage Channel <R©C) and provided programmable PME 
control/status signal I/O via the FTIQ channel. The ETK> channel is the 32-bit bus which Interconnected the 
ISP to the etgnt adapters. The ETIO channel reled on a custom interface protocol wth was supported by 

69 hardware on the ETIO adapter end software on the ISP. 
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Problems addressed by our APAP machine 

The machine which v*$ have called the Advanced ParaBel Array Processor (APAP) 19 a fine-grained 
peraflel processor wbwh wo beoeve is needed 10 address issues of prior designs. As illustrated above, there 

3 have been man/ fine-grained (and also coarse-grained) processors constructed from both point design and 
oftehe-ahelf processors using darJcstad and shaved memory and any one of the many possible Intercon- 
nection schemes. To dan these approaches have all encountered one or more design and performance 
limitations. Each "solution 1 " leads in a different c&ectfon. Each has problems. Existing parallel machines 
are difficult to program. Each 19 not generally adaptable to various sizes of machines compatible ecross a 

70 range of applications. Each has its design limitations caused by physical design, interconnection and 
eretiitectuiel issues. 

Physical tejugS 

J3 Some approaches utilize a separata chip design for each 07 the various functions required in a 
horizontal structure. These approaches suffer performance fimlEaflons due 10 chip crossing delays, 

Other approaches Integrate various functions together verScafly inso a single chip. These approached 
suffer performance Imitations due to the physical limit on the number of toglo gates which can be 
Integrated onto a producible chip. 

Interconnection Issues 

Networks which imerconnect the various processing functions are important to fine-grained parallel 
processors. Processor designs with buses, m&sh&e, and hyparc<rf>es have all been developed. Each of 
these networks has Inherent Srnrtebons as to processing capability. Buses Srnlt both the number of 
processors which can be physically infftjOTtnscfed and the nemo* performance. Meshes lead to large 
network diameters wrifch limit network performance. Hypercobes require each node to have a large number 
of interconnection pons; the number of processors which can be t reconnected 1$ limbed by the physical 
incwtoutput pifts at the node, Hypertubea are recogntzed as having some significant p^r^mance gains 
30 over ihe prior bus and mesh structures. 

Arehfociurel Issues; 

Processes which are suftabte tor tine-grafcned parallel processors faS into t**o cfistiiKi types- Processes 
38 which are functionally partieonable lend to perform better on multiple instruction, mu&ipte data (MIMD) 
architectures. Processes which are no! functionally- personable but have multiple data streams tend to 
perform belter on single instruction, multiple data (&1M0) architectures. For any given appicatton, there is 
fikefy to be some number of both types of processes. System tade-orTs are required to pick me 
architecture which best suits a particular application but no single solution has been satisfactory. 

40 

SUMMARY OF THE INVEHTfOM 

We have created a new way to make massively paraJei processors and csher computer systems by 
creating a new "chip" and systems designed with our new concepts. This application is directed to such 

« systems. Components described in our eppacattons can be combined in our systems to make new 
systems. They also can be combined wfth existing technology, 

TOnfc, our tette CMOS DRAM chip of approximately n x 14 mm can be put together much like bricks 
are walled in a building or paved to form a or** road. Our chip provide* the structure necessary to build a 
"house", a complex computer system, by connected replication. 

30 Racing our development In perspective, four little chlpsu each one alike, each one with eight or more 
processors enrtoedd&d in memory wSh an internal array capability and external VO broadcast and control 
interface, would provide me memory and processing power of lhirty-sw or more complex computers, and 
they could all be placed with compact hybrid packaging into something the size of a watch, and operated 
with very low power, as each chip only dissipates about 2 wara. With this chip, we have creased many new 

55 concepts, and those that we consider our invention are described in detail in the description and claims. 
The systems that can be created with our computer system can range from small devices to massfrve 
machines with PETAOP potential. 
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Our little memory chip array processor vr* call our Advanced Parafla* Array Processor. Thou^i small, ri 
complex and powerful. A typical cluster will hove many cRIps. 
Many aspect and features erf invention have been described in and related eppficstio*s. 
concepts and daturas of invention improve and ara applicable to computer systems which may noi employ 

5 each inventor*. We believe our concepts and features win be adopted and used In tfie next century. 

This technical description provides an overview of our Advanced PsraSei Array Processor <APAP} 
representing our new memory concepts and our effort In developing a scalable massively para del processor 
(MPP) that Is simple (very small number of unique part numbed and has very high performance Out 
processor unfizes in Its preferred embodiment a VLSI chip. The chip comprises 2n PME mferocornputers. 

3o "n" represente the maximum number of array dimensionality. The chip further comprises s broadcast and 
control interface (BCD and internal and externa] communication pa*» between PMEs on the chip among 
themselves and to the on onto system environment The preferred chip has 8 PMEs (but w* also can 
provide mere) and one BCT The 2n PMEs and DC I are considered a node. This node can function in eifear 
51MB or WIMD mode, in duet $IMQ/MODE> with asynchronous processing, and with SIMIMD functionamy. 

is Since it is scalable, this approach provides a node which can be the main building block for scalable 
parallel processors of varying size. The microcomputer architecture of the PME presides fully distributed 
message passing interconnection and control features within each node, or chip. Each node provides 
multiple parapet microcomputer capabHBy at the chip level, the rrweroprocesor or personal computer levef, at 
a worksteilen level, at special application levels which may be represented by a vision and/or avionics level, 

to and, when fully extended, to capability at greater tevefe with powerful QfeafJop performance into the 
supercomputer range. The simplicity is achieved by the use oi a single highly extended DRAM Chip ihat is 
replicated mm parallel clusters. This Keeps the part number count down and allows scaling cepabifity to the 
cost or performance need, by varying fra chip count, then ihe number of modules, etc. 

Our approach enables us lo provide e maohlne with etsributes meetfng the requirements that drive to a 

29 parallel solution In a series of applications, Our methods of peraSelizetlon at fro sub-chip level serve to keep 
weight, volume, and recurring and logistic costs down. 

Because our different size systems are all based upon a single chip, software toots are common lor all 
sbe systems. This offers the potential of development software (running on smaller workstation machines) 
that is interchangeable among ell tevefc (workstation, aerospace, and supercomputer). Thai advantage 

so means programmers can develop programs on workstations while a production program runs on a much 
larger machine. 

As a resuft of our well befenoed ctewgn implvmentsraon we meet today's requirements imposed by 
technology, performance, cost and perception, and enable growth of the system Into the future. Since our 
MPP approach starts at the chip level, our discussion starts at the chip technology description and 
» concludes with the supercomputer application descriptions. 

Physical iftterconnection. and architecture! issues *w all be addressed In the machine directly. 
FuncSons w» not only be integrated into a single chip design, but the chip desiQn wlB provide functions 
sufffciemry powerful and flexible ihai the chip will be effective at processing, routing, storage and three 
classes of I/O. The interconnection network, vvlfi be a new version of the hypercube which provides minimum 

6 new/orfc diameters without the inputtoutput pin and -wireetfity imitations normally associated wsh hyper- 
cubes. The trade-off between 3>MD and MIMD ace eliminated because the design allows processors lo 
dynamically switch between MIMD end Simd mode. "iffia eliminates many problems which will be 
encountered by application programmers of 'hybrid* machines, fen addfflon, design wW snow a subset erf 
the processors ta be in SIMD or MIMD mode. 

<%> me Advanced Parallel Array Processor (apap) is a fine-groined parallel processor, it consists of comroi 
and processing sectfons which ape pertHionabte such that configuration* tufteble for superccmptrimg 
through personal oompuiing appJkietione can be saUefieti. In most configurations it ^ould attach to a host 
processor end support the off loading of segments of the IW* weiWoed- Because toe APAP array 
processing eternente am genets! purpose computers, the pattariar type of worktoed off-loaded will vary 

so depending upon the capabilities of the host For example, our APAP can be a module tor an IBM 3090 
vector processor mainframe. When attached to a mainframe wrth high Derforrnsnce vector floating poiffi 
capability the task off-loaded might be sparse to dense matrix transformations. Alternatively, when attached 
to a PC personal computer Ihe off-loaded task might be numerically intensive 3 dimensional graphics 
processing. 

sb The above referenced parent USSN 07/611 ,584, filed November 1600 of Diettenderfer ©t aL, tided 
''Paraflci Associative Processor System" describes the idea of integrating computer memory and control 
logic wRhin a single chip and recreating the combination within the chip and building a processor system 
out of replications of the single chip. This approach which ie continued and expanded here leads to a 
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system which provides massively parallel processing capability at the cost of developing end manufacturing 
only a single chip typo white enhancing performance capability by reducing the chip boundary crossings 
and fine length. 

The ebgve rerorenced parent US8N 07*1 filed November 13, 1080 Hltfrtrated wlftradon of f- 

s dlmsns tonal I/O s(ructures(essentlaBy a linear VO) vv&h multiple SJMD PMEs attached to that structure within 
a chip. This embodiment elaborates tnese concepts *> dimensions greater men 1. Tns description which 
toltows wlil De In terms of ^-dimensional HQ structures with 8 SIMD-M1MD PMEs per chip. However, that 
can be extended fc> greater dlmensionn&ty or more PMEs per dimension as v*e will describe with respect to 
R31IRES3, 8, 10. 15 and 16. 

10 Our processing element Includes a full fO system including both date transfer and program Interrupts. 
Our description of our preferred embc<ftme<it will be primarily described in terms of the preferred 4- 
dimensional i*> structures with D SIMD/MIMD PMEs per chip, which has especial advantages now in our 
view. However. thai can be extended to greater meristonaffly or more PMEs per dimension as described 
In our parent eppficauon. in addition, for most applications we prefer and have made inventions fri areas of 

is greater dimensions with hypercube intsrconnections, preferably with the modified hypercube we describe. 
However, In some applications a 2-dimenstonaJ mesh mlsrconnection of chips will be applicable to a task at 
hand For instance, in certain misery compute** a 2 cftmeasione! mesh will be saitabfe and cost effective. 

This disclosure extends the concepts from the fnfcerprocoseor communication to tho external 
Input /Oulput fecHttles and describes the Interfaces and modules required lor control of the processing array. 

20 In summary three types of TO. inier-procesaoi. processors to/from external, end DroadcesVoontroi are 
described. Massively parcel processing systems require all ties© types erf tt> bandwidth demands to be 
balanced with processor computing capability. vVtihin the array these requirements win be satisfied by 
replicating a 16 bit (reduced) instruction set processor, augmented with very fast interrupt state swapping 
capability- Thai processor is referred to as ihe PME iltostraling the preferred embodiment ol our APAP. The 

2S characteristics of ihe PME ere completely unique when compared wfch the processing dements en other 
massively parens! machines, rt permits big processing, routing, storage and I'D to ba comptetely distributed. 
This is not characteristic of any ether design. 

in a hypercube each PME can address as its neighbor, any PME whose address differs in any single bit 
position. In a ring, any PME can address as m neighbor the two PMEs whose aofrsssee differ a i. The 

so modified hypercube of our preferred embodiment utilized for the APAP combines these approaclrcs by 
building hypercubes ouJ of rinos- The intersection of rings is denned to be a node. Each node of our 
preferred system has its PME, memory and I'Q. snd oSw features ef the node, formed in a semiconductor 
silicon tow level CMOS DRAM chip. Nodes are constructed from muJUple PMEs on each chip. Each PME 
exists in only one rbig cf nodes. PMEs wHhn> node are connected by editions) rings soch that 

as communications can be routed between rings within the node. This leads to the addressing structure where 
any PME can step messages toward the objective by addressing a PME tn Hs own ring or an acSaceni PME 
wShin the node. In sssence a PME can address a PME whose address differs by i in one in the 1ns d bit 
field ol its ring (where d is aha number of PMEs in (he nng) or the PME with Ihe same address but existing 
in an adjacent dimension. The PME effectively appears to exist In n sets of rings, while in ecuielty It exists 

40 only in one red ring and one hidden ring rotetiy contained wttin the chip. The dimensionality for me 
modified hypercube is def bed to be the vat us n from the previous sentence. 

We prefer to use a modified hypercube. This Is elaborated in the part of this application describing the 
technology. Finally, PMEs win in a ring am paired sucn tnat one moves data externally clockwise along a 
ring of nodes and the other moves data externally counterclockwise along ite ring of nodes* thus dedicating 

4$ a PME to an external port. 

In our massively parallel machine, in our preferred embodiment ihe interconnection and broadcast of 
data and Instructions from one PME to another PME In the node and externally of the node to oBter nodes 
of a cluster or PMEs of a massively parallel processing environment are performed by a programmable 
router, allowing reconfiguration and virtual flexfltfrty to the network operations. This important feature is fully 

so distributed and embedded in the Pfcflr and allows for processor communication and data transfers among 
PMEs during operations of the system in SIMD and MMO modes, as well as in the SIMDrMftlD and 
SPvfflVlQ modes of operabon- 

WitNn ihe rings each Interconnection teg is a pomhto-point connection. Each PME has a polnKo-polnt 
connection with the two neighboring PMEs in its ring and with two neighboring PMEs in two adjacent rings, 

es Three of these point-to-point connections are interna! to the node, while the fourth point- to-poir* connection 
is to an adjacent node. 

Tne massively parallel processing system uses the processing elements, with their local memory and 
interconnect topology to conned all processors to each other Embedded within the PME is our fully 


It 


EPO 570 729 A2 


distributed I/O programmers router. Our system also providss an addition to the system which provides ihe 
ebfllty to toad and unload all the processing elements. With our 2>pper we provide a method lor taetfng end 
untoedmg of tl>e array of PEs and thus enable *mpi&rr.erttatkjn of a test NO along an edge of the awe/a 
rings. To provide for external interface CO any subset of the rings may be broken 4un-zipped across some 

s cftmenstan(s» with the resultant broken paths connected to Hie externa] Interface. The co-pending applica- 
tion entitled "APAP I/O ZIPPER", filed concurrsnay herewith, USSN . fried May 22.1992. 
describes our 'Tipper* in actional detail The 'zipper' can be applied to only the subset of links required to 
support the peek external I/O load* which in ell configurations considered so fa* feeds to Its being appfied 
onry to one or two edges of the physical design, 

to The final type of VO consists of dela that must be broadcast to, or gaftered from all PMEs. plus date 
which is too soecialized to fit on the standard buses, Broadcast data Includes commands, programs end 
data. Gathered date is primarily stofus and monitor tancttons vrttfs diagnostic and test functions are She 
specialized elements. Each node, in addition to the included set of PMEs. contains one Broadcast and 
Control Interface (BCI) section. 

is Consider PMEs rrierconnected in a modified 4 dimensional hypercube network. If each ring contains 18 
PMEs, (hen the ay stem wis have 32.738 PMEs. The network diameter la 18 steps. Each PME contains In 
S/W the router and recofioguretion s/w to support a particular outgoing port. Thus, software routing 
provides the capability to reconfigure m the event ot a faulty processing element or node. Inherent in a ^d, 
25 Mhz network design vlih byte wide half duplex rings to the provision for 410 gigabytes per second peek 

so Internal bandwidfr 

The 4 dimensional hypercube Jsads to a particularly advantageous package. Bghi of the PMEs 
(Including data ftw l memory end I/O paths and controls) are encompassed in a single chip. Thus, a node 
will be a single chip including pairs of efemeitis along the rings. The nodes are configured together In an 8 
X 8 array lo make up a duster. The kilty populated machine is built up of an ooay of 0 X 8 clusters to 

ss provide the maximum capacity of ^738 PMEs. 

Each PME is a powerful microcorfiputer having significant memory and 1*0 functions. There is mutiibyte 
data flow within a reduced instruction set {RISO architecture Each PME has 16 bit internal dale flow and 
eight levels of program Interrupts wtih the use of working and general registers to manage date flo*. There 
is a circuit switched and store and forward mode for I/O transfer under PME software control The SIMD 

so mode or MIMD mode Is under PME software control. The PME can execute RISC Inetruottone from either 
the BCI in a SIMD mode, or from its own main memory in MIMD mode. Specific RISC instruction code 
points cen be reinterpreted to perfonm unique functions in the SIMD mode. Each PME can implement en 
extended Instruction Set Architecture and provide routings which perform macro level Instructions such as 
extended precision fixed point arithmetic, Boating point aitihmetic, vector arifrmetic, and the l&e. This 

76 permits not only complex mam to be handled but image processing activities jot display or image date in 
multiple dimensions (2d and 3d images) and for multimedia applications. The system can select groups of 
PMEs for a function. PMES assigned can eltocaie selected data, and instructions for group processing The 
operation© can be extemaDy monitored via the 601. Each Ed has a primary control inpui, a secondary 
control Input and a status monitor output lor She node. Within a node me 2n PMEs can be connection for a 

40 binary hypercube ccrrtmurticsiion network within the chip- Communication between PMEs is controlled by 
She bits in PME control registers under control of PME software. This permits the system to have a virtual 
routing capability. Each PME can step messages up or do#n Its own right Of to Its neighboring PME to 
either of tyro adjacent rings. Each interface between PME* Is a point-to-point connection. The 00 porta 
permit otT-chfp extensions ot Bis internal ring to adjacent nodes of the system. Trie system is buat up of 

* replications of a node to form a node array, a cluster, and other ccrflguralicm 

To cornpfernsnt our system's SIMD, MWD, and SIMIMD functionality, our development we 

have provided unique operational modes. Among our SIMD/MIMD PME's unique modes are new 
functional features reined to aa .he "store and toward / circuit switch" functions. These hardware functions 
complemented with the on chip cemmunkation and programmable Internal and externa UO rouBng provides 

so die PME with very optimal data transferring capability. In preferred mode of operation the processor 
memory is gsrwafly the date sink for messages and date targeted a? the PME in the store and forward 
mode. Messages and data not targeted for the PME are sent directly to the required output pore when In 
circuit switched mode. The PME software performs the selected routing path while giving the PME a 
dynamically selectable store and toward t circuit switch functionality, 

53 Among fhe advances vte frwr*e provided is a fully distributed architecture for PMEs of a node, Each 
node has 2n processors* memory and VO. Every PME will provide very flexible processing capability with 
16 bit data Host, S4K bytes of local storage, store end fc<we/d*ircwr switch logic, PME to PME 
communication. SIMO/MMO switching capabilities, programmable routing, and defeated floating point 
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assist logic. The organization of every PME and Its communication psihs v^ith other PMEs within the same 
chip to minimize chip crossing delays. PME functions can be indopendontly oporaiod by the PME and 
integrated with functions to fre node, a duster; anci larger arrays. 

Our massively parallel system is made up of nodal building blocks of muBi processor nodes, clusters of 

s nodes, and arrays of PMEs already packaged In clusters. For control of these packaged systems vshe 
provide a system array director which with the Iwrdwera oontroflara performs the overall Processing 
Memory Element iPME) Array Controller functions in the massively parallel processing environment The 
Dhector comprises of three functional areas, the Application Interface, the Cluster Synchronizer, end 
normally a Ctueter Controller. The Array Director vrtt bcve the overall control of the PME ©nay, using the 

70 broadcast bus and our zipper connection to steer data and commands to all ol the PMEs. The Array 
Director tactions as e software system interacting with the hardware to perform the role as the shell of the 
APAP operating system. 

The interconnection for our PMEs lor a massively parallel array computer SIMD/M1MD processing 
memory element (PME> interconnecton provides the processor to processor connection In the massively 

jo peralleJ processing errvrronmeni Each PME uiitizes our fully distributed interprets ear communication 
hardware from ins on-chip PME to PME connection, to the off-chip I/O facilities which support the chip-to- 
chip interconnection. Our modified topology limits our duster to cluster wiring vrhBe supporting me 
advantages of hypercube connections, 

The concepts which we employ (or a PME node are related to the VLSI packaging techniques used for 

20 the Advanced Peraflel Array Processor (APAP) computer system disclosed here, which packaging features 
of our invention provide enhancements to the manufacturing abirty of me APAP system. These techniques 
are unique m the area of massively parallel processor machines and wai enable the machine to be 
packaged and configured in optimal subsets that can be bull and tested. 

The packaging techniques lake advantage of rhe eight PMEs packaged in a single chip and arranged in 

as a N-olmensional modified hypercube configuration. Tnis chip level package or node of the array toe 
smallest building block in the APAP design. These nodes are men packaged in an 6 x S array where the +- 
X and the ^-Y makes rings within the array or cluster and the *»W, and *-Z are brought out to the 
nei#*ortng dusters. A grouping of clusters make up an away. The intended applications for APAP 
computers depend upon the particular cortftgursaon and host Large systems attached to mainframes with 

so effecQve vectorised floating point processors might address special vectoriaable problems - such as weather 
prediction* wind tunnel airnuJaiion, turbulent fluid modeling and finiie element modeling. Where those 
problems involve sparse matrices, significant worfr must be done to prepare the data for vectorized 
arithmetic and IBcewlse to store results. That workload vrould be off loaded to the APAP, In Intermediate size 
systems, the APAP might be dedicated to perfbnning trie graphics operations associated vvriti visualization* 

as or wlft some preprocessing operation on Incoming data 0.e n porfemang optimum assignment problems in 
miKtery sensor fusion appHc&8ons). Small systems attached to workstations or PCs might ser»e ae 
programmer development staaons or roighi emulate a vectorial floating point processor attachment or a 
3d graphics processor. 


<0 BR1BF DESCRIPTION OF THE DRAWINGS 

HO. 1 show a parallel processor processing element like those which would uSlze old technology. 

FIG. 2 ehowg a massively paraJei processor buSding block in accordance wtfh our Invention, 
representing our new chEp design. 
**> HQ. 3 illustrates on the right sfete the preferred cwp physical cluster layout for our preferred 
embodiment of a chip single node fme grained paralte processor. There each chip ts a 
ecaSabJe ps/eJIel prooeseor chip providing 5 fcflPe performance vtfth CMOS DRAM memory 
end logic permiftieg e> cooled implemejiteiion of massive concurrent systems. On the left 
dd© of Figure 3, tfwre is illustrated the replaced technology, 
so HG. 4 shows a computer processor furtcflonaJ block diagram In accordance with the Invention. 

FIG. & shows a typical Advanced Parallel Array Prooeseor computer system configuration. 

R<5. 6 shows a system overview of our ana-grained parallel processor technology In accordance 
with out invention, Illustrating system build up using replication of She PME element which 
permits systems to be developed with 40 to 193,8«0 MI'S performance. 
55 FIG. 7 illustrates she hardware for the processing element (PME) data flow and local memory in 
accordance wHh our invention, while 

RG. 8 illustrates PME data now where a processor memory element is configured as a hardwired 
general purpose computer that provides about 5 MFS fixed point processing or A Mflope via 
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programmed control floating point operators 
FIG. 9 shews the PME to PME connection (binary hypercutoe) and dota pains that can be taken in 
accordance with OlM invert tfiO'\ wh3e 

FIG. 10 Kusfrstes node interconnectons for the chip or node which has 8 PMEs. each of which 
s manages a single external port and permits distribution of Uie network control function end 

eliminates a functional hardware port boiaeneck. 
Fid, II Is a Week diagram of a scalable parcel processor chip where each PME Is a 16 bit wide 
processor Ndth 32 K words of local memory and there Is \f0 porting for a broadcast port 
which provides a ccrrtrolter-to-^l Interface white external porta are tWNctiorwl poIrcHo-potrtf 
70 interfaces permitting ring torus connections within the chip and ex ternary. 

Fi-3. i2 shows an array director in the preferred embodiment 

FIG. 13 in part (a) illustrates me system bo3 to or ton a cluster array coupling, embtina feeding or 
unloading of the array by connecting the edges of duelers to the system bus (see FIGURE 
14). In FIGURE 13 In pert (b3 there is the bus to/from the processing element portion. 
*5 FIGURE t3 illustrates how muttipl* system buses can be supported with multiple dustera. 

Each cluster can support 50 to 57 Mbyte's bandwidth. 

RG 14 shows a w zEppe« ,r connecnon for fast VO connection. 

FIG. 13 shows an 0 degree hypercube connection Hlustfatlng a peckeglng technique in accordance 
wltfi our invention applcable Id an 8 degree hypercube. 

20 FIG 1 0 $bow$ two independent node connections in tfre hypercube. 

FIG. 17 shows the Bitonic Sort dgoviftm as an example to iiiustrara the advantages of the defined 

Simd/mimd processor system. 
FIG. id illustrates a system bloc*; diagram for a host attached large system wan one application 
processor interlace iBu&traled. This illustration may also be viewed with the understanding 

29 that our Invention may be employed In stand alone systems which use multiple application 

processor interfaces. Such interfaces in a FIGURE 18 configuration wffi support 
QASD&raphlcs on all or many clusters. Workstation accelerators can eliminate the host, 
application processor snterface (API) and cluster synchronber <CS) Illustrated by emulation. 
The OS & not required in ail instance 

ao FIG. 19 llustrates lie software development environment for our system. Programs can be prepared 
by and executed from the host application processor. Soft program and machine debug is 
supported by the workstation based consote illustrated here and m FIGURE 22. Both of these 
services will support applications operating on a real or a simulated MM P. enabling apoSce- 
tions to be developed at a workstation level as well as on a supercomputer tamed of tl>e 

as APAP MMP. The common software environment enhances programmabiOty and distributed 

usage. 

FIG. 20 ilustretes the propj&nmmg levels which are per mHted by me new systems. As different 
users require more or lees detailed knewtodgo, the software system is developed to support 
mis venation. At the highest level the user does not need to know the architecture Is indeed 
<** an MMP. The system can be used with existing language systems for partifiorteig ot 

programs, such as parallel Fortran- 

FIG. 21 Ilustretes the parallel Forttan compiler system for ihe MMP provided by the APAP conSgura- 
ifons described A sequential to parallel compiler system uses a combination of existing 
cornpiier capability with new data allocation functions and enables use of a partitioning 
45 program r&e FoarsnD. 

FIG. 22 illustrates the worfcstzrdon application of me APAP, rvhere the APAP becomes a wkstertion 
accelerator. Note tfat the unit has the same physical size aa a FUSC/8000 Model 530, but 
mis model now contains an mmp *«ch is attached to the woAstetion via a bus extension 
module Illustrated. 

so FIG. 23 llustrates an application for an APAP MMP module for en AW ACS mBHary or commercial 
application. This is a way of handing efNcien^y ihe classical distributed sensor fusion 
problem shown in FIGURE 23, where fte observation to tack matching is ciassicaJfy done 
with YveB know algorithms like nearest neighbor* 2 dimensional Inear assignment (Munkes-). 
probabilistic data association or multlpte hypothetic testing, buf these can now be done In an 

65 improved manner as frustrated by FIGURES 2d and 25. 

FI3. 24 ilustretes how the system pioddes *>e abfflty to handle n-diroensiooai assignment problems 
in real time, 

FIG. 25 illustrates processing flow for an rvdtmensional assignment problem utita'ng an APAP. 
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RG 26 illustrates the expansion unit provided by the system enclosure described showing how a 
uftii can provide 424 MflopS or 5120 MIPS using only 8 to 10 extended SBWE modules, 
provide ih© p$rfo*TA€inc0 comparable to that of specialized signal processor module in only 
cub» feet This system can become a SIMD massive machine wfch 1024 parallel 
e processors performing two bllBon operations per second (G0PS> end can grow by adding 

1024 addnionaJ processors end 32MB additional storage 
HO- 27 illustrates ine APAP packing for a supercomputer. Here is a large system of comparable 
performance but much ampler footprint then other systems. H can be built by replicating the 
APAP cluster within an enclosure like those weed for smaller machines, 
70 We have provided, as part of the description. Tables ttustrating As hardwired instructions for a P ME. in 
which Table 1 illustrates Fteed-porrn arithmetic instructions: Table 2 illustrates storage to storage inatruc- 
iions; Table 3 MJustrafts logical instructions; Table 4 iltasfrstes shjif instructions; Tatoe 5 illustrates branch 
hstruoiions; Table 6 illustrates the statue swrfching instructions; and Table- 7 illustrates the inputfbutpul 
instructions* 

rs (Note: For convenience of MbsiraSon rn the formal patent drawings, ROUF&3 may be separated in parte 
and as a convention we place the top of the FIGURE as the first sheet with subsequent sheets proceeding 
down and across when viewing the FIGURE, in the event thai multiple sheets are used} 

Our detailed description fotbws with parte explaining the preferred embooSrnente of our mvenbon 
provided by way of example. 

DETAILED DESCRIPTION OF THE IrWENTION 

Turning now fo our Invention in greater derail, it *viB be seen from FIGURE I, which illustrates trie 
existing technology level, illustrated by (ho transputer TOOO chip, and representing similar chips for such 

24 machines as the illustrated by *e Touchstone Delta 00OQ. M Cube ( dm and others When FIGURE 1 te 
compared with foe devetorxnerits here, it m» be seen mat not only can systems like the prior systems bs 
substantially Improved by employing our invention, but also new powerful systems can be created as we 
win describe. FIGURE 1's conventional modern microprocessor technology consumes pfcns and memory. 
Bandwidih is limited and inter-chip ccttmuntaaon drags the system down. 

ao The new technology leapfrog represented by FIGURE 2 merges processors, memory, ID into multiple 
PMEs (eighf oi more 16 bit processors each oi which has no memory access deteys and uses ell *e pins 
for networking) formed on a single low power CMOS DRAM chip. The system can make use of ideas of our 
prior referenced cfieclosures as weB as Invention separately described In the applications filed conciirrenay 
herewith and sppliceble to the system we describe here- Thus, for this purpose they are Incorporated herein 

as by reference. Our concepts or grouping, autonomy, trariBparenoy, zipper interaction, asynchronous SIMD, 
SIMIMD or SlMLVMIMD. can all be employed vdfci (he new technology, even fcough to lesser advantage 
they can be employed in the systems of the prior technology and m earrta'naiiori v/fth our own piior muttrpis 
picket prooossor. 

Our picket system can employ the present processor- Our basic concept is that** have now provided 
40 a replfcablft brick, a new basic twitting block for systems 7/iih our new memory processor, a memory unit 
having embedded processors, router and VO. This bask: building block is scalable. The basic system which 
we have tmplemented employs a 4 Meg. CMOS DRAM. It Is expendable to be used in larger memory 
conflguraitons, ?4th i&Mblf DRAMS, and e4MDJt chips by e>paastoa Each processor Is a gate array. Wlfr 
denser deposition, marry more processors* at higher clock speeds, can be placed on the same chip, and 
using sates and additional memory m expand the performance of each PME. Scaling a single part type 
provides a system framwork and architecture which can have a performance wed into the PETAOP range- 

FIGURE 2 illustrates the memory processor which we call the PME or processor memory element in 
accordance with our preferred embodiment The processor has eight or more processors. In the pictured 
embodiment there are eight The chip can be expanded (horizontally) to add more processors. The chip 
so can. as preferred, retain Ute logic and expand the DRAM memory with additional ce3s Bnearly (vertically). 
Pictured are 16 - 32k by 9 tat sections of DRAM memory surmundrng a field of CMOS gate em*y gates 
which implement & replications of a id bit wide data flow processors. 

Using IBM CMOS low power sub-micron BM CMOS deposition on silicon technology, ft uses selected 
etlFCon wen trench to provide significant storage on a smas chip surface. Our memory and multiple 
66 processors organized interconnect is made with IBM's advanced art of making semiconductor chips. 
However tt wilt be recognized that the wie chip we describe has about A Meg. memory, it la designed so 
ihas as 16 Meg. memory technology becomes stable, when improved yields and methods of accommodat- 
ing defects are certain, our little chip can migrate k> larger memory sizes each £ bits wide wttfiout changing 


EP0 573 72SA2 


the logic. Advances m photo and X-ray ffihography keep pushing minimum feature size to well below 5 
microns. Our design erMsJons more progress. Those* advances will permit placement of very (urge amounts 
of memory with processing on a single silicon chip. 

Our device is a 4 MEG CMOS DRAM believed to too fcrst general memory chip with extensive room 

5 lor logic. 18 replications of a 32k by 9-blt DRAM macro make up the memory array. The DRAM has 120K 
cells it allocates wifti sigraficsirt surface area for application logte on the chip, with triple level metal wiring. 
The processor logic cells are preferably gaw array ceils. The 35 ns or less DRAM access time marches the 
processor cycle time. This CMOS Implementation provides logic density for a very effective PE (picket) end 
does so while dissipating 1.3 walls for the tonic The separate memory sector* of the chip, each 3£K by 9 

jo bite, expansion noi change© logic) surrounds the field of CMOS gate array gales representing 120K 
ceil* and hashing the logic described to other figures. Memory is barriered and with a separated povfcsr 
source dissipates 3 waits. In providing the combining of tfgmcmi amounts or logic on the same silicon 
substrate win significant amounts of memory problems involved with the electrical noise incompatibtSty of 
logic and DRAM have been overcome. Logic tends to be very noisy tttrie memory needs relative quiet to 

?5 sense the millivolt size slgnais that result Irom reading the cells of DRAM. We prefer to provide trenched 
fripto msfed layer fill con deposition, with separate barriered portions of the memory chip devoted to memory 
and lo processor logic with voltage aod ground isoiaaon, and separeie power disfribufioo and barriers, to 
achieve compatfbffty between logic and DRAM. 

3D APAP System Overview of Prefeiied Emoodbnems 

Tins description introduces the new technology tn too following order: 

1. Technology 

2. Chip H.W description 

6 3. Networking and system buiW up 

4, Software 

5. Applications 

The initial sections of the detailed description describe how DRAM low power CMOS chips are made 
to Include S processors on and as part of w& manufactured PME DRAM chips each supporting; 
a* f. 16 bit 5 NIP dataflows, 

2, independent instruction stream end interrupt processing and 

3. 8 bit (plus parity and corrirds) wide external port and interconnection to 3 oiher on chip processors. 
Our invention provides multiple luncUons which are Integrated into a single chip design. The chip will 

provkk PME Rinctions which powerful and flexible and feifficieitfy so such that a chip having ecslsbrfty 
33 will be effective at processing, routing, storage and three clesses of IflCX This chip has integrated memory 
and corwi logic within the single chip w mate fse pme« and this comtHnelien is repealed wtthio ihe chip. 
A processor system Is bull from replications of trie single chip. 

The approach partitions the km power CMOS DRAM. It wiH be termed as muftpte word length <1 8) bit 
by 32K secSons, associating one socmen vrftfi a processor- (We use the term PME to refer to a single 
processor, memory and I/O capable system ur8) This psrftionmg leads to each DRAM chip being an B 
way 'cubs connected' MIMD parallel processor with & byte vride independent interconnection ports. (See 
FIGURE e for an illustraeon of a replication of fine-grained paraiel technology, fiJustraang replication and the 
ring torus posslbfiates.) 

The software description addresses several distinct program types. At the lowest level, processes 
<* imertace the user's program (or services celled by the application) to the detailed hardware H/w needs. 
This level includes the tasks required to manage the K) and interprocesser synchrc™ ratten and is vmei 
might be called a microprogram for the MPP. An intermediate level of services provide for both mapping 
applications ^developed wtsh vector or matrix operations} to the MPP, and ateo control, synchronization, 
startup, dagnosfc functions. At the host level, high order languages are supported by library functions that 
so support uecfotzed programs with either simple automatic data aSocatfon to the MPP or user tuned data 
aflocattoa The multi-level software SrW approach permits apportions to expioft different ctegrses of control 
and optimlzsrJon within a single program. Thus, a user can code appi cation, programs without understanding 
the architecture detail while an optimizer might tune at the microcode level only fte small high usage 
kernels of a program. 

ss Sections of our description that describe 1024 element 5 QIPS units and a elemerrt 1G4 GiPS 

unit Illustrate the range ot possible systems. However* those are not the limits; both omafler and larger units 
are feasible. Tress particular sizes have beta selected as examples because the small unit fa suitable to 
microprocessor? {accelerators), personal computers, workstation and mltaary applications (using of coarse 

ie 
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different packaging techniques* while the larger unit b IHustraSve of s mainframe application as a module or 
complete supercomputer system. A software description will provide examples of ether challenging work 
that might be efocifvely programed on each of the illustrative systems. 

9 PME DRAM CMOS - A BASE FOR A MULTIPROCESSOR PME 

FIGURE 2 illustrates our technology improvement at me chip technology level. This exisndabte 
computer oigenization la very cost and performance efficient over the wide of system steea because 
It uses only one chip type. Combining the memory and processing on one chip eimfnatea the pine 

70 dedicated to the memory bus and their associated reliability and performance penalties . Repfcatbn of our 
design within the chip make* it economically feasible to oonsWer cusorn logic designs for processor 
subsections- Replication of the chip w3hin the system leads to large scale manufacturing economies. 
Final*/, CMOS technology requires low power per MIP> which in turn minimizes power supply and cooling 
needs. The chip architecture can be programmed lor multiple word lengths enabling operations to be 

js performed ftat would otherwise require much larger length processors. In combination these attributes 
permit the extensive range of system performance. 

Our new technology can be compared wrih a possible extension of me oW ieciwotogy it overlaps. It ie 
apparent thai the advantages of smaller teafejres have been used by processor designers to construct more 
complex chips and by memory designers to provide greater replication of the simple element. If the trend 

so continues one could expect memories to get four times as large -while processors might exploit density to: 

1. Include multiple execute units with instruction routers, 

2. increase cache sizes and essoctetfce cape&Hity and/or 

3. increase instruction look ahead and advance computation capability. 

However, these approaches to (he old technology illustrated by FIGURE 1 all tend to dead end. 

so Duplicating processors leads to linearly increasing ptn requirements but pine per chip Is fixed. Better cache- 
ing can only explo* the application's data reuse pattern. Beyond mat, memory bandwdS) becomes the Sma. 
Application data dependencies and branching Bmft too potential advantage of look ahead schemes. 
Additionally, it is not apparent that MPP applications wlm finegrained paraflelism need 1, 4. or 18 
Me$aword memories per processing unit Attempting, to share such large memories between munipto 

as processors resuto In severe memory bandwidth limitations. 

Our new approach is not dead ended. We? combine bom significant memory and WO and ptocessor into 
a single chip, es Hlustrated by the FIGURE 2 and subsequent illustration and description. It reduces pert 
number requirements and eliminates the delays associated with chip crossing. More Importantly, this 
permits all the chip's HO pins to be dedicated to intorproceasor communteation and thus, maximizes 

as network bandwidth. 

To implement our preferred embodiment iBustreied In FIGURE 2 we use a process U»at is available now, 
using IBM low power CMOS technology. Our illustrated embodiment can be made wKh CMOS DRAM 
density, in CMOS and can bo implemented in denser CMOS. Our illustrated embedment of 32K memory 
cells tor each of 8 PWEs on a chip can be increased as CMOS becomes denser. In our embodiment we 
*o utifce the real estate and process technology for a 4 MEG CMOS DRAM, and expand mis with processor 

replication associated with &2K memory on the chip fcaetf. The chip, it wis* be seen has processor, memory , 
and W m each of the chip packages of the cluster shown in FIGURE 3. Within each package is a memory 
wfth embextosa processor element, router, ana Ml all certain** in a 4 MCG CMOS DRAM belayed to be 
the first general memory chip w&h extensive room for logic. II uses selected siHcon wit* trench to provide 

4s significant storage on a small chip surface. Each processor chip of out design aaemativefy can bs made 
wfth 10 rep&caoons of a 32K by d bn DRAM macro ns) using JS? micron CMOS logic to make up the 
memory array. The device to unique ?n thai U allocates ourf ace area for 1 20 K cells of application logic on 
the ch~{>, supported by the capacity of triple level metal wiring. The multiple cards of the old technology Is 
shown crossed out on the left side of FIGURE & 

so Our basic nepBcable element brick technology Is an answer to the old technology, ff one considered the 
"Xed 1 * technology on the left of FIGURE 3, one would see too many chips, too many cards, and waste. For 
example, Today 1 * proposed teratlop machines thai others offer would have tftersity a mMon or more chtpe In 
them. Wlft todays other technology only a few percent oJ these chips, gt best are tiuty operations 
producers. The mat are "overhead 11 (typicany memory. neMork interface, etc,>- 

56 it w9l become evident that it is not feasible to package such chips, in such a large number, in anything 
that must operate in a constrained environment of physical stee* {How many could you ft* in a small area ol 
a cockpa?} Furthermore, such proposed terafop machines o* others, already huge, must scale up 1 0OGx 
times to reach the petaop range. We have a solution which dramatically decreases the percent of non- 
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operations producing chips. We provicte increased bandvvkfih. We provide this wrthin a reasonable network 
(SmertttonaBty. With such a brick technology, where memory becomes the operator, and networks are used 
for passing controls* where operations producing chips are drameticeHy increased. In addition* the upgrade 
dramatically reduces the numbs* of different types of chips. Our system is designed for scale-up, without a 

s requirement lor specialized packaging, cooling, power, or environmental can strain tsL 

W*h our brick technology, uaiizino instead of separate processors, memory units wtft bum in 
processors and network capability, tie configurason shown in FIGURE 3, representing a card, with chips 
which ere pin competed with current 4Mblt DRAM card* at the connector level Such a single card could 
hold, with a desk* point of a baste 40 mfc per chip oerlorrrience leva, 32 chips, or 1280 mips. Four such 

70 cards would provide 5 gips. The vrorkstaiion configuration which t& illustrated woupd preferably have such a 
PE memory array, a duster controller, and an IBM RISC System/6000 which has sufficient performance to 
run and monitor execution of an array processor application developed at me wortsti&an- 

A very gate efficient processor can be used in Ihe processor portion. Such designs for prooeescrs have 
been employed, but never within memory. Meed, In addition, we have provided Ihe ability to mix MWD 

» and SIMD bask: operation provisions. Our chip provides a "broadcast bus* which provides an alternate path 
teiio each CPU's instruction buffer. Cur eitastar controller issues commands to each of the pes in the PMEs, 
and these can be stored in the PME to control their operation in one mode or another. Each PME does not 
have to store an enire program, but can store only those portions applicable to a given task at various 
limes during processing of an appscaflon. 

to Given the basic device one can elect to develop a single processor memory combinati&ri. Alternafveiy , 
by usfeig a more simple processor and a subset of ihe memory macros one can design for either £. 4, 0 or 
t$ replications of ihe basic processing element <pme^ The pme can be made ampler either by adjusting 
the dataflow bandwidth or by substituting processor cycles for functional accelerators. For most embodi- 
ments wa prefer lo make 8 replications of Ihe basic processing element we describe, 

as Ow application studies have indicated that for no* the most favorable answer is 6 replications of a i$ 
bit wide daia flow and 32K word memory. We conclude this because: 
1 . 16 bit words permit single cycle fetch of Instructions and addresses. 

2. 8 PMEs each wfcth en external port permits 4 dimensional torus Interconnections. Using 4 c* 8 PMEs 
on each ring leads to modules suitable ier fte range of targeted system performance 
3D 3, 8 external ports requires about 50% of ihe ohip pins, providing sufficient remainder for power, ground 
end common control signal 

4, S Processors Implemented in a 64 KB/te Man Store 

a. allows for a register based arcWiecttire rather than a memory mapped amlilecrure, and it 

b. forces some desirable but not required accelerators to be implemented by mgfiipfe processor 
as cycle©. 

This lest attribute is important because it permits use of the developing logic density increase- Our new 
accelerators (ex. floating point aithmetic unit per PME> are added as crisp hardware without affecting 
system design, pins and cables or application code. 

Trie resultant chip layout and size (14.59 * 1433 mm) is shewn tn FIGURE 2, and FIGURE 3 shows a 
duster or such chips, which can be packaged in systems like those stow* in later FIGURES sor eland alone 
unite, workstations which elide next to a workstation host with a connection bus, in AWACs spptwaiJone, and 
in supercomputers* This chip technology provides a numbet of system level advantages, it permits 
development of ihe scalable MPP by basic replication of a single part type. The two DRAM macros per 
processor provide sufficient storage for both data and program. An SRAM of equivalent size might consume 

<w more man 10 times more power. This advantage permits MDMD rnacNre models rather than the more 
limited SIMD models characteristic of machines with single chip processor/memory designs- The 35 ns or 
teas DRAM access time matches Ihe expected processor cycle time. CMOS logic provides the logic density 
for a very effecfive PME and dees so while dissipating only 1.3 watts. (Total chip power is 1.3 + .9 
(memory; = Z2 Thorn features In turn permit using tsie chip in MR. applications requiring conduction 

so cooling. (Air cooling in non-ftf L appicatkms is significantly- easier.) However, the air cooled embodiment can 
be used for workstation aj>d other environmental. A ersndalone processor mr$tt be configured vrifa an 80 
amp - 5 volt power supply. 

Advanced Parallel Array Processor <APAP) bullrjng toe* 5 are shown lo FIGURE 4 and In FI-3URE 5. 
FIGURE 4 illustrates me iunctfonal block csagram of me Advanced Paraaei Array Processor. MuRiple 

65 oppfcatjon interfaces 150, 160, 170, 180 exist for the application processor 1 00 or processors 1 10, 150, 
13Q. FIGURE 5 Illustrates the baste building Weeks that can be configured into different system block 
CHagrams- The APAP, in a rnaximum configuration, can incorporate 32,768 identical PMEs, The processor 
consists of the PME Array 280, 200. 300, 3l0> an Array Director 250 and an Application Processor Interface 
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£63 for the appEcsiian processor 200 or procussore 210, 220, 230. The Array Director 255 consists of thrws 
ftmcfJonaJ units: Application Processes interface 260. duster Synchronizer 270 and duetar Coniroaer 270. 
An Array Director can perform the functions of th© array controller of our prior linear picket system for SIMD 
operations with MIMD capability. The cluster control &r 270, along with a eot of 64 Array clusters 230, 290. 

5 300. 310. (le. duster or 512 PMEs). is Ihe basic building block or the APAP computer system. The 
elements of m Array Director 250 permit configuring systems wft & wide range of cluster repltaatbns. This 
modularhy baaed upon strict replication of both processing and control elements Is unique, to this massively 
parallel computer system. In add-on, the Application ftwessor interface 200 supports the Test/Debug 
device 240 which vtf i accomplish important design, debug, and rr<onaoring functions 

10 Control lore ore assembled wtih a wott-defined interface, eg, IB Ms hticrochanoel, used in other systama 
today, including controllers wish 800 processors. Held programmable gate arrays add functions to the 
controller whicti can b& changed to meet s partlcutar conligur^jion's rscjuiremsnts (how many PMEs then* 
are. flterr couplings, etc) 

The PME arrays 260. 290> 300> 310 contain t*> functions needed to operate as ettter SIMD or MIMD 
/s devices. They aiao contain fundiona thai permit the complete aei or PMEs to be divided irrbo 1 to 258 
distinct subsets. When divided into subsets the Array Director 250 Interleaves between subsets. The 
sequence of the rnterfeave process and the amount of control exercised over each subset Is program 
controlled. This capability to operate distinct subsets of the array In one mode. I.e., MIMD with differing 
programs. white oilier sets operate In tightly synchronized SIMD mode under Array Director control* 
to represent an advance in the art- Severe! examples presented later iDu^rate the edveniegea of tt^e concept. 

Array Architecture 

The set of nodes forming tte Array is connected as a rvdirnension&l modified hyper cube. In that 
£S Interconnection scheme, each node haa direct connections to 2n other nodes. Those connections can be 
either simplex, halNduplex or fun-duplex type paths. In any dimension greaser than 3d, the modified 
hypercube Is a new concept In Interconnection techniques (The modified hypercube in the 2d case 
generates a torus, and in the 3d case an orthogonally connect latace wtih edge surfaces wrapped to 
opposing ew face.) 

so To describe the interconnection scheme for greater than 3d cases requires an inductive description. A 
set of mi nodes can be interconnected as a ring- (The ring could be 'simply connected 1 , 1 breioVd , i T c*oss 
connected', 'fally connected*, etc Although additional node ports ore needed for greater frsn simple rings, 
that added complexity does not affeci the modified hypercube ai/udure.) Tlie rra rings can then be linked 
together by connecting each equivalent node in ihe set of rings. Tim resut at this point is a Jorus. To 

98 construct a i + Id modified hypercube from an id modified hypercube, » sets of id modified hypercubes 
and Interconnect el of the equivalent m ( - level nodes into rings- 

Trus process is iaoQtFated tor the 4d modified hypercube. using mj « 8 tori - i ~4 by me illustration in 
FIGURE S, Compare our description under node Topology and also R3UR6S 6, 0, 10, 15 and 16, 

FIGURE 6 illustrates the fine-grained parallel technology path from the single processor dement 300, 

<o mads up of 32K ie-bit words with a 16-Wt processor to the Network node 310 of eight processors 312 and 
then- associated memory 3f I wifo their Mty distributed I/O routers 313 and Signal tO ports 514, 315, on 
through groups of nodes labeled dusters 320 and Into the cluster configuration 360 and to the various 
application* 330, 340, 350, 370. The 2d level structure is the duster 320, and 64 dusters are Integrated to 
form the 4d modified hypercube of 32.708 Processing Elemente 360. 

Processing Array Element (PME) Preferred Embodiment 

As IHustraled by F10URE 2 and FIGURE 11 the preferred APAP has a basic buitfog block of a one 
chip node. Each node contains 0 identical processor memory elements (PMEs) and one broadcast and 
so control interface (BCi). white some of our inventions may be Implemented when all functions are not on the 
sama chip, r? is important ironri a per formence and cost reduction standpoint to provide the chip as & one 
chip node wfth the £ processor memory elements using the advanced technology which we have described 
and can be Implemented today. 

The preferred implemsffi&iion of a PME has a 6<i KByte main store, 1 6 10-Wi general registers on each 
55 of fi program interrupt levels, a full function anThmetic/logte unit (ALU) vwBi working registers, a statue 

register, and four r^ograrnmabte bKfiredton^ VQ ports, in add-on the purred implementation provides a 
SWD mode broadcast interface via the broadcast and control interface (BCI) vmich allows an external 
comroller (see our original parent application and the description of ow currently preferred embodiment for 
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a nodal array and system with dusters) to drive PME operation decode, memory address, and ALU data 
Inputs. Thte chip can perform the function* or a mtonMompuier allowing multiple paraSai operations to be 
performed witfiin it, and it can be coupled to other chip* within 9 system of multiple node©, whether by an 
BrorconnootiDn network, a mesh or hypercube network, or our preferred and advanced scalable embodi- 
s rrvent 

The PMEs are Interconnected In a series of rings or tori in cur preferred scalable embodtaterri. In some 
applications the nodes could be Werconnected In a mesh. In our preferred embodiment each nods contains 
two PMEs In each of four tori. The tori are denoted wjc/v, and z (see FIGURE 6). figure 11 deplete the 
rrrfercorfwctton of PME3 within a node. The two PMEs in each torus are designated by their external HD 
jo porH+trV, -W, +X. -X, +V. -V f +Z. -Z), Within Ihe node, there are aSso two rings which interconnect the A 
4 n and 4 -n PMEa. These Internal rings provide the path for messages to move between the external tort 

Soce ihe APAP can be In our preferred errJborfmem a four dimensional ortrwgonal array, the internal 
rings allow messages to move throughout the array in ail dimensions. 

The PMEs ere self-contained stored program microcomputers comprising a main store, local store, 
'9 opsraacn decode, sitthnrietfclogtc unit (ALU), working registers and Input/Output VO ports. The PMEs have 
the capability of fetching and executing stored Instructions from dier onn main store bi hUMD operation or 
to fetch and execute commands via the Bd alter face fan SIMD mode. This interface permits Hnercemmimi- 
csiion among the controller, the PME, and other PMEs in a system made up of mufaplo chips, 

The BCi is the node's Interlace to the external array controller element and to an array dreetor. The 
20 BCi provides common rode functions such as timers end docks. The bci provides broadcast function 
masking for each nodal PME and provides ihe physical Interface and buffering tor ihn broadcest-busnD- 
PME data iransters, and also provides Che nodal Interface to system status and monJiorlng and debug 
elements. 

Each PME contains separate interrupt levels to support each of its point-to-point interfaces and the 

29 broadcast interface- Data Is Input to the PME main store or output from PME main store under Direct 
Memory Access (DMA} control. An "input transfer complete* interrupt is available for each of the interfaces 
to signal the PME software that data is present. Status information ts available for the software to determine 
the completion of data output operations, 

Each PME has a "circuit switched mode 1 ' of VO In which one oi rtt four input poris can be switched 

3D directly to ones of tts four output ports, without having the data enter ihe PME main store. Selection of the 
source and desfinaSon of the "circuit s^nch* is under control of *» software executing on the PME Tlie 
other three input ports continue to hare access to PME main store functions, white fro fourth input is 
switched to an output port. 

An addreonaJ type of I/O has data that roust be broadcast to, or gathered from ail pmes, plus data 

3s which is too specialized to fit on the standard buses. Broadcast data can intiude SI MO commends, MIMD 
programs, and SIMD data. Gathered data la primarily status and monitor Mictions. DLagnosttc and test 
functions are the specialized data elements. Each node, *1 addition to tl» included set of PMEs, contains 
one 601- During operations the 601 section monitors the broadcast interface and stoers/colfoctB broadcast 
data tottom the addressed PMEfs). A combination of enabRng masks and addressing tegs are used by the 

so BCI to determine what broadcast information la intended tor which pmes. 

Each PME is capable of operating in 3f4D or in MIMD mode in our preferred embodiment In SIMD 
mode, each instruction is led Into the PME from the broadcast bus via the BCI. The BCI buffers each 
broadcast data word urr$ aU or Its selected nodal PMEs have used it This synchronization provides 
accommodation of the data liming dependencies associated with the execution of SIMD commands and 

«s aiowe asynchronous operafcons to be performed oy a pme. in wbivtd mode, each pme executes its own 
program from Its own main store. The PMEs are Initialized to ths* SIMD mods. For MWD operations, the 
external controller normally broadcasts the program to each or the PMEs while they are in SIMD mode, and 
then commands the PMEs to switch to MWD mode and begin executing. Masking/tagging the broadcast 
information allows dbfemrt sets of PMEs to contain different MIMD programs, and or selected sets of PMEs 

50 to operate in MIMD mode srhlle other sets of PMEs execute In SIMD mode. In various software clusters or 
partitions these separate functions can operate independently of the actions in other clusters or partitions. 
The operation of the Instruction Set Architecture (ISA) of the PME will vary slightly depending on whether 
the PME Is In the SIMD or MIMD mode. Most ISA faistrucBons operate Ictenfcceiry regardless of mode. 
However, since the PME In SIMD mode doss not perform branching or other control functions acme code 

85 points dedicated to (hose MIMD instructions are reinterpreted in StMD mode to aQow the PME to perform 
special operations such as searching main memory tor a match to a broadcast date value or switching to 
MMO mods. Tors further extends system flexibility of an may. 
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P WE Architecture 

B$elcaJly : our prefeired architecture comprises a PME which has a 10 bH wide data flow, 32K of 16 bit 
memory, sporiaiteod I/O pore and L O swtohing paths, plus the necessary control logic to permft each PME 

5 to fetcb. decode end execute the 13 bil tnstrucUon set provided by our instruction set architecture (ISA). 
The preferred PME performs ft* fuocfcms of a virtual router, find thus performs both the processing 
functions and data router functions. The memory organization allows by cross addressing of memory 
between pme? access to a large random access memory, and direct memory for the PME. Tho individual 
PME memory can be aS local, or divided into local and shared global areas programmsacai^y. Specialized 

10 comrols and capabilities which we describe permit rapid task switching and retention of program state 
wormatk>n at each of tho pmes inwrupl executfoo levels Although some of the capabilities we provide 
have existed in other processors, thsir appBcaHcn for management of imerpnoceeeor I.O is unique in 
massively parallel machines. An example is the integrate of the message router function into the PME itself. 
This elrnlnetes epectaflaerj router chips or development of specialized VLSI routers. We also recognize that 

js in some instances one could distribute the functions we provide on a single chip onto several chips 
interconnected by metafization layers or otherwise and accomplish Improvements to massively parallel 
machines- Further, as our architecture is scalable from a single node to massively paraBeJ supercomputer 
level machines, ft is possible io utiftte some or our concepts at dffieront levels. As we will Bluslrate lor 
example our PME data flow Is very powerful and yel operates to make (he scalable design effective, 

so The PME processing memory element develops for each of the rouble PMEfc ot a node, a Wy 
distributed architecture. Every PME wis be comprised of processing capability with 16 bit data flow, 64K 
bytes of local storage, store and forward/circuit switch logic PME to PME communication. Simd/mimd 
switching capabilities programmable routing, end decfieatsd Hcs&mo point assist logic. These functions can 
be independent operated by the PME and integrated with oilier PMEs within Uho seme chip to minimise 

20 chip crossing delays. Referring to FIGURES 7 and 8 we illustrate the PME dataflow. The PME consists of 
16 Off w«3e dataflow 425. 435, 445. 455, 465. 32K by 16 bit memory 420, Specialized I/O porfe 400. 410, 
480, 400 and I/O switching paths 425, ptuc the necessary control logic to permit the PME to fetch, decode 
and execute a 16 bit reduced instruction set 43a 440> 450, 480. The special logic also permits Ihe PME to 
perform as both the processing, unit 460 and data router. Specialized controls 405, 40&\ 407. 408 and 

3D capabtilUes are Incorporated to permit rapid task switching and retention of program state Information at 
each of the pmes* infeirupi execution levels. Such capabilities have been included in other processors; 
however, their application specrHcaity for management of Interprocessor UO is unique in massively parallel 
machines. Specifically, it permits the integration of the muter function into the PME without requiring 
specialized chips or VLSI development macros. 

38 

16 bit internal data flow and control 

The major parts of tho interna* data flow of tho processing dement are shown in FIGURE 7. FIGURE 7 
illustrates the internal data flow of the processing element. Thb processing element has a full 19 bit infernal 

40 data flow 425. 43% 445. 455, 461 The Important paths of the internal data flows use 12 nanosecond hard 
registers such as she OP register 450, M register 440, WR register -170, and the program counter PC 
register 430. These registers feed the luBy distributed alu 490 and I/O router registers and logic 405. 40s, 
407, 408 for all operations. With current VLSI technology, me processor can execute memory operations 
and instruction steps at 25 Mh2> and it can build the important elements, OP register 460, M register 440, 
wr regis** 470> and the PC register 430 wtft 12 nanosecond hard registers. Other reared registers are 
mapped to memory locations. 

as seen in FIGURE 8 (ha internal data now of the PME has ita 3ZK oy 16 bit main store in the form ot 
two dram macros. The remeindei of the dele flow consists of CMOS gste array macros. AM of the memory 
can be formed wift the logic wfch low power CMOS ORAM deposition techniques to form an very large 

so scaled Integrated PME chip node. The PME Is repBcated 8 times In tho preferred embodiment of the node 
chip. The PME data flow consists of a 1$ word by 16 bit genera? register stack, a muRHuncnon 
aFtthmstiologJc unit (ALU) working registers to buffer memory addresses, memory output registers, ALU 
output registers, operation/command, I/O output registers, and muWpfews to select inputs lo the ALU and 
registers. Current CMOS VLSI technology for 4MByte DRAM memory with our logic permits a PME to 

ss execute instruction steps at 25Mh*. We am providing fte OP register, ihe M register, the WR regbtar and 
the general register stack wltb 12 nanosecond hard registers. Other required registers are mapped to 
memory locations kvIWd a PME, 
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The PME data fkw b designed as a 16 bit integer arithmetic processor. 8pedai multiplexor pstfhs have 
been added to optimise eL'trcutme emulation of n x 18 bU floating point opsrattona (n=>1). The 16 bit data 
flow permits effect emulation of floating point epilations. Specific pat« within she date flow h** been 
included to permit floating point operations in as littb as 10 cycles. The ISA Sndudes special cede point to 

5 permit subroutines for extended (longer than Id-bit} operand operations. The subsequent floating point 
performance te approximately one twentieth the fixed fleeting point performance, Thfe performance is 
adequate to eliminate the need for special floating point chips augmenting the PME as Is characteristic of 
other messfaely parallel machines. Some other processors do Include the spec**) floating point processors 
or> the same chip a* a single proceeeor (See FIGURE 1>. We can enable special fleeting point hardware 

70 processors on the earns chip with our PMEs but we would now need additional ceHs than is required tor the 
preferred embodiment. For floating point op©rarjons k see also the concurrently filed FLOATING POihTT 
application referenced above for imprownente to the IEEE standard. 

The approach developed is wbD poised lo take anVantage of the norma] increaeee in VLSI technology 
performance. As circuit size shrinks and greater packaging (tensity becomes possible then data flow 

>s (dements like base and index registers, currently mapped to memory could be moved to hardware. 
Likewise, f baling point sub-steps are accelerated with additional hardware which vra wll prefer for the 
developing CMOS DRAM technology as reliable higher density levels aie achieved. Very importantly, this 
hardware alternative does not affect any software. 

The PME Is Initialized to SJMD mods with interrupts disabled. Commands are fed Into me PME 

20 operation decode buffer from the BCI. Each ?me en instruction operation completes, the PME requests a 
new command from the BCl in e similar manner, immediate data is requested from the BCI et the 
appropriate point In the Instruction execution cycle. Most instructions of the ISA operate identically whether 
the PME i* in SIMD mode or in MIMD mode, wrft ft* exception of that SIMD instruction* and immediate 
data are taken from the BCI; m fcPMD mode Iho PME maintains a program counter (PC) end uses (hat as 

29 the address within & own memory to fetch a 13 Wt instruction. Inductions such ee "Branch" which 
expifcffly address the program counter have no meaning in SIMD mode and acme of ftose code points are 
reinterpreted to perform special £WD functions as comparing immediate data against an area of main store 

The PME instruction decode logic permits either SIIVTO/WIMD operational modes, and PMEs can 
tcansition between modes dynamically, in SIMD mode the PME received decoded induction information 

30 and executes that data In foe next clock cycle. In MIMD mode to PME matnlatns a program counter PC 
address and uses that as fbe address vrithtft ris own memory to fetch a 1$ bn instruction. Instruction decode 
and execution proceeds as in most any other ftSC type machine. A PME in SiMD mode enter? MIMD 
mode when gteen the Information lhai repreeents a decode branch. A PME In MIMD mode enters Ihe SIMD 
mode upon executing a specific instruction for the transition. 

35 When PMEs transition dynamically between SIMD and MIMD modes, an MIMD mode is entered by 
executton of a $IMD "write control register* instruction with rhe appropriate control bit set to a r i At the 
completion of the SIMD ststruction, m PME enters the MWD mode, enables intenupta, and begins 
fetching and executing its MIMD inefruciions from the main store location specified by its general register 
R0. Interrupts are masked or unmasked depending on the state of interrupt masks when the MIMD control 

^ bit is set. The PME returns to SIMD mode eaher by being externally reinitialized or by executing a MIMD 
"•writ© control register" instruction wrtft the appropriate control bit set to aero. 

Data communication paths and control 

w Returning to Figure 7 it win be seen that each pme has 3 input ports 400, and 3 output ports 480 
intended for on-chip communication plus 1 I/O port 410, A$0 for off chip communications- Existing 
technology , rather than Ao processor idee; require* (hat the off-chip perl be byte wide half duplex. Input 
ports are connected such that data may be routed from Input to memory, or from input AR register 405 to 
output register 408 via direct 16 ba data path 425, Memory would be the date sink wr messages targeted at 

so the PME or for massages that were moved in 'store and forward' mode. Messages ftel do not target the 
particular PME are sent directly to fte required output port, providing a 'circuit switched' mode, when 
blocking has not occurred The PME S/V/ rs charged with performing the routing and determming the 
selected transmission mode- TWe makes dynamically selecting between 'circuited switched* and 'store and 
forvratP modes possible. This is also another unique characterise of the PME design. 

55 Thus, our preferred node has fi PMEs and each PME has 4- output porta (Left. Right, Vertical, and 
External}. Three of the Input pete and throe of the output porta are 16-bit wide hill duple* point-to-point 
connections to the other PMEs on the chfr The fourth ports are combined m the preferred enrolment to 
provide a half duplex point-to-point ccnnecSon to an off-chip PME. Due to pin end power constraints that we 
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have imposed to mate use of the less dense CMOS we employ, ths actual of teMp interlace is a byle-wkte 
path which ta used to multiplex two halves of the lnter-Phffl= data word. With Bpedal "ripper" 1 drcuJiry which 
provides a dynamic, tempos iogJcal breaking of iniermodal rings to allow data to enter or lew an array, 
these Qxtemd PME ports provide the APAP oxramaJ I/O arrsy function. 

0 For data routed to the PME memory, normal DMA is supported such (hat the PME Instruction stream 
must become Involved in the VO processing only at me beginning and end of massages. FinaSy, data that ie 
being 'circuit switched to an Internal output pom Is forwarded wfehout docking. This permits single cycte 
data transfers within a chip and detects when chip crossings vdi! occur such that the fastest but still refleWe 
communicatori cen occur. Fart tawwflng rtUze* forward data paths and backward control paths, eJI 

w operating in transparent mode. In essence, a source looks through several stages to see the acknowledg- 
ments from fte PME performing a DMA or ofKttp transfer. 

As se*n by FIGURES 7 and 8 Data on a PME input port may be destined tor the local PME, or for a 
PME further down the ring. Data destined for a PME further down the ring may be stored in the local PME 
main memory and then forwarded by the local PME towards the target PME {store and forward), or the local 

ss input port may be logically connected ro a pertlcufev local output port (circuit switched) such that the data 
passes ^transparently'' through lhe local PME on its way to the target PME. Local PME software 
dynamicaBy co^trola whether or not the local PME is in 'store and forward" mode or in ^circuit switched 1 ' 
mode tor any o? the four inputs and four ouspute. rn circuit switched mode, the PWE concurrently processes 
all functions except the I/O associated wish (he circuit switch; In store and forward mode the PME suspends 

20 all other processing funcoc-ft* begin the irQ Awarding process. 

While data may be stored externa^ of the array in a shared memory or DASO (with external controller), 
rt may be stored anywhere ir> the memories provided by PMEs. Input data destined for the local PME or 
buffered in m local PME during *store and toward" operations is placed into local PME mam memory via 
a direct memory access (address} mechanism associated widh each of the input ports, A program interrupt 

£$ Is available to indicate that a message has been loaded into PME main memory. The local PME program 
interprets header data to determine it the data destined tor the local PME te a control message ^icn can 
be used io set up a drcuii-swi&ched paift to another PME, or whether it is a message to be forwarded to 
another PME. Circuit switched paths are conhoEed by local PME software. A circuit switched path k&c&ty 
couples a PME input path directly to an output p&h wfihout passing through any intervening buffer storage. 

30 Since the output paths between PMEs on Hie same chip have no Intervening buffer storage, data can enter 
the chip, pess through a number of PMEs on the chip and be loaded info a target PMPs main memory in a 
single clock cycle! Only when a circuit switch combination leaves the chip, fs an intermediate buffer storage 
required This reduces tie effective dtanrieter of the APAP array by a number of unbuffered circuit switched 
paths. As a resutt data can be sent item a PME to a target PME in as taw clock cycles as there are 

as intervening chips, regardless of the number of PMEs in the pain. This kind of routing can be compered to a 
switched endiwmem In which at each node cycles are required to carry data on to the next node. Each of 
oui nodes has 8 PMEs! 

Memory and Interrupt Levels 

The PME contains 32K by 1 & bit 426 dedicated storage words. This storage is completely general and 
can contain both data and program. In &MD operations all of memory could be data as is characteristic of 
other SIMD massively parallel machines, in MIMD modes, me memory is quite normal; but unite most 
massively parallel MIMD machines fris merr>ory is on the same chip vrifft the PME and is thus, immediately 
available- This men eliminates the need for cache-frig and cache coherency sechnkjues characteristic of 
other massively parallel MIMD machines. In the case for Instance of tamos's chics orrfy 4K resides on the 
chip, and external memory interface bus and pins are required. These are eBrrinated by us. 

Low order storage locations are used to provide a set of general purpose registers for each interrupt 
level. The particular ISA developed for the PME uses short address fields for these register references. 

so Interrupts are utilized to manage processing, I/O activities and SAV specified functions (i.e. , a PME in 
noimal processing will switch to an Interrupt level wr>&n incoming vo iniiktes). if tl)* level & not masked, 
the svrltch is made by changing a pointer fri HW such that registers are accessed from a new section of 
low order memory and by snapping a single PC value. This technique permits rest level switching and 
permits SW to avoid the normal register save operations and sjso to eave status wlthJn the interrupt ravel 

55 registers. 

The PME processor operates on one of eight program interrupt levels. Memory addressing permHs a 
par&fcring of me fewer 576 words of memory amoung the eight levels of interrupts, of these 578 v*ordte 
of memory are directly addressable by programs executing on any of tie eight Jevets. The other 512 words 
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are pertWoned Into eight 64 word segments. Each 64 word segment Is directly accessible only by programs 
executing on its associated Interrupt level. Indirect addressing techniques ate employed for allowing all 
programs to access *H 32K words of PME memory. 

The interrupt levels are assigned to the input ports, the BCl, and to error handling. There is a *normd" 
s level but there Is no "privileged" nor "supervisor leveL A program interrupt causes a context switch In 
svhfcn the oonfersa of the PC program counter, stalusrcontrol register, and selected general registers are 
stored in speerrted main memory locations and new values for these registers are fetched from other 
specified main memory locations. 

The PME data flow discussed with reference to FIGURES 7 and 8, may be amplified by reference to 
so the additional sections below. In a complex system, the PME data flow uses the combination of the chip as 
an array node with memory* processor and I/O which sends and receives messages wish the BCt that we 
replicate as the basic buteSng block of an MMP bum wit) our APAP, The MMP can handle many wort 
lengths. 

is PME Mufcjpje Length Data Row Processing 

The system we describe can perform fba operations handled by current processors wB* the dasa flow In 
the PME which la 16 bits wide. This done by performing operation? on date lengths which are multiples 
of 18 bits. This Ls accomplished by doing the operation In 13 bit pieces. One may need to know the result 
39 of each piece fi e. was ft zero, was there s carry out of the high bfc9 of Ihe sum). 

Adding two numbers of 46 bits can be an example of data flow. In this example two numbers of 40 bite 
( af(M7) and W0-47) ) are added by performing ffle following in the hardware: 

a#2«47) + r^"32-47h>enB<32-47> step one 

1) save the carry out of high bit of sum 

2) remember if partial result was aero or non-zero 

a> a(16-31 } + W 6-31 ) + save carry<>ang(1&41) step two 


1) save the carry out of high bit of sum 

2) remember If partW result was zero or non-zero from this result and from previous step; If &o$h are 
os zero remember zero; if either is non-zero remember non-aero 

a(f>15) + b(f>l5j + saved carry->ene(C-1 5) final step 


4$ 1) if ifsa piece ie zero andi&i piece was zero ana is zero 

2) if ihte piece is zero and last piece was non-zero an? rs non-zero 

3) if this place is non-zero ana is positive or negative based on sign of sum (assuming no overflow) 

4) ti carry into sign of ans os not-equal to carry out of sign of answer, ans has wrong sign and result is 
an overflow £can not properly represent in the avaHable btis) 

45 The length can be extended by repeating the second step m the rnwie as many times as required, if the 
Isngth wsre 32 the second step would not be performed. If the lengti were greater than 48, step two would 
be done multiple limes. If lha length ware just 16 the operation in step one, with conditions 3 and 4 of the 
final step would be done. Extend no, the length of the operands to multiple fenojia of the data 3ow is a 
technique having a consequence that the Instruction usually taxes longer to execute for a narrower data 

so flow. That Is. a 22 bit add on a 32 bU data flow only takes one pass through the adder logic, while the seme 
add on a 1 6 bit data flow lakes two passes through the adder logic. 

What we have done that to interesting re that in the current kmplementabon of the machine we have 
single Instruction* which can perform add^subtra^c^ycompfuea/moves on operands of length 1 to 8 words 
(lengft fa defined aa pan of the instrucfan). individual instructions aveaable to the programmer perform the 

95 same kind of operations as shown above for steps one, two. and Real (except to the programmer the 
operand length la longer i.e. 19 to 128 bits), At the bare bones hardware level, we are working on ie bits at 
a lime, but a>3 programmer thinks sme's doing 1$ to 128 bits at a ttrne. 
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By using combination r of these instructions, operands of any length con be manipulated by ths 
programmer Le. two instruaoons can be used lo add two numbers of up to 263 bits In lengttt 

PME Processor 

5 

Our PME processor different from modern mtaorxoceasors currently utiSzed for MPP applications, 
Ths processor portion rJBITerences include: 

1. the processor 15 a fufly rxtjgremmabie hardwired computer {see the ISA desertion for an instruction 
set overview} with: 

ro o ft has a complete memory module shown in the upper right comer {see FIGURE Qy, 

0 * has hardware regfcfcrs wtth controls required to ©mutate separate register sets for each interrupt 

level {shown fci upper teft corner), 
o iSB ALU has ihe required registers and controls to permit effective mulHjrycle integer and floating 

point arithmetic, 

/s o 9 has I/O switching paths needed to support packet or circuit switched data movement between 

PMEs Interconnected by point-to-point links shown In (he lower right comer, 

2. This is ow iranimeJ-tat approach to processor design permhttng eight repftcatlons of the PME per chip 
with the CMOS DRAM technology. 

3. This processor portion of the PME provides about the minimum dataflow width required to encode a 
& fast Instruction Set Architecture {ISA) -see Tables - which i$ required to permit effective MIMD or SIMD 

operation of our MMP, 

PME Resident Software 

£0 The PME is the smallest element of the APAP capable of executing a stored program ft can execute a 
program whkh is residem in som3 external control element and ted to it by the broadcast and control 
interface <GCi> in SIMD mode or it can execute a program wbkfc te resident In Its own main memory {MIMD 
mode)- it can dynamically switch between SIMD mode and MHWD mode representing SIMEVMIMD mode 
dusfty tactions, and also the system can execute these ciuaBfjae at the same isme {SIMIMD mode). A 

30 particular PME can make this dynamic switch by merely setting or resetting a btl In a control register. Since 
SIMD PME software id actually resident in the external control element further discussion of this may be 
found in our discussion of the Array Director and in related applications* 

MIMD software ia stoned Into the PME main memory while the PME is in SIMD mode. This Is Feasible 
since many of the PMEs wil contain identical programs because frey wiB be rxoceaemg similar data in an 

as asynchronous manner. Here vto would not© that these programs are not fixed, but ihey can be modified by 
loading the MIMD program Irom an external source during processing of other operations. 

Since the PME N^truction set archfteciure represented in the Tables that of a microcomputer, there 
are tew restrictions with this architecture on Uhe functions which the PME can execute. Eesenliaiy each 
PME can function Sice a RISC microprocessor. Typical MIMD PME software routines are Sated below: 

40 1. Basic control programs for despatching and pdortttemg the various resident routines, 

2. Communication software to pass data and control messages among the PMEs. This software would 
determine when a particular PME would 50 into/out of the "circuit switched" mode, ft performs a ^tore 
end forward" function as appropriate, it also initiates, sends, receives, and terminate* messages between 
its own main memory and that of another PME. 
4S 3. interrupt handling software completes the context switch* and responds to an event which has caused 
tie treerrupt These can indude fart-safe routines and rerouting or reassignment of PMEs to an array, 

4. Routines which implement the extended instruction Sal Architecture which we describe beta*. These 
routings perform macro level instructions such as extended precision fixed point srtihmetjc, floating point 
arift melk:, vector arithmetic, and the lite. This permits not only complex main to be handled but image 

so processing activities tor display of image data fcn multiple dimensions {2d and 3d Images) and multimedia 
processes, 

5. Standard rnafoemaifcal Horary functions can be included. These can preferably Include LiNPAK and 
VPSS routines. Since each PME may be operating on a tffleretH element of a vector or matrix, the 
various PMEs may *£ be executing different routines or differing portions of the seme matrix at one ilrne, 

58 6. Specialised routines for performing scaisertyilher or sorting functions which take advantage of ths 
APAP nodal irterccrmactron structure and permit dynamic muW^ftmenslonal routing are provided. The 
routnes effectively ta*e advantage cf some arnount of synchronkaSon provided among the various 
PMEa, while permitting asynchronous operations to cominte. For sort*, there are sort routines. The 
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APAP is wbB sufed to a Batcher Sort. Because that sort requires extensive calculations to determine 
particular elemanl K> compare veraus very short comparison cycles. Program oyhGhronizaSon is man- 
aged by the I/O Statements. The program allows multiple data elements per PME end very large parefel 
sorts in quite a straic^it forward manner, 
s White each PME has Its own resident software. Hie systems made from these nricrocomputers can 
axecufe higher level language processes designed to scalar and parallel machines. Thus the system can 
execute operation programs vtfilcb have been written tor UNIX machines, or those of other operating 
systems, in high level languages such as Fornm C, C + + . FortranD. and so on. 

e may bo an Interesting footnote that our processor concepts use an approach to processor design 
w which is quite old. Perhaps thirty years of us© of a similar ISA design has occurred in IBM's military 
processors. We have been the *st to reoognke that this kind of design can be used to advantage lo 
leapfrog the dead ended current modem microprocessor design when combined with our total PME design 
to move Ihe technology to a new path lor use in trie next century. 

Although the processors design characteristics are quite different from other modern microprocessors, 
>5 simitar gate constrained military and aerospace processors have used the design since the '80s, It provides 
sufficient instructions and registers for straight forward compeer development* and both general and signal 
processing applications are effectively ruaning on this design. Our design has minimal gate requirements, 
and ©M has implemented some similar concepts for years when embedded chip designs were needed 
general purpose processing. Our adoption now or parts of flie older ISA. design permits usa of marry ul Utiles 
to and ofrer software vehicles which wW enable adoption of our systems as a rapid rats because of the 
existing base and the knowtsdgpe tfiai many programmers have of the design concepts. 

PMEK) 

aj The PME wfll interface to me broadcast and control Interface (BCl) hue by either reading data from the 
bus into tie ALU via the peth labeled BCl in FIG WE 8 or by f aching insertions irom the bus directly into 
(he decode loglo (not chown). The PME powers up in &MD mode and in thai mode reads, decodes and 
executes Instructions until encountering a branch. A broadcast command SIMD mode causes the transition 
to MIWID with Mwtruceons fetched tocs&y. A broadcast PME instruction 'INTERNAL DrOW 1 reverts the state. 

30 PME I/O can be sencfng data* passing data or receding data. When sending data the PME sets ihe 
CTL register to connect XJVfflT to eitter L> R> V, or x. HW services then pass a block of data from memory 
to the target via the ALU multiplexer and the XMIT repstor- This processing interleaves with normal 
Instruction operation. Depending upon application requirements, the block of data transmitted can contain 
raw data for a predefined PME anoVor commands to establish paths, A PME mat receives data win store 

as Input to memory and Interrupt tie active lower level processing. The interpretation task at the interrupt level 
can use the interrupt event to do task synchroniiaUcn or initiate a transparent I/O operation (when data to 
addressed elsewhere.) During f>e transparent I/O operation, me PME is iree to continue execution. Its CTL 
register makes it a bridge, Data will pass through it without gating, and it will remain in that mode until an 
Instruction or the data stream resets CTL While a PME Is passing data It cannot be a data source*, but ft 
can be a data sink tor another meseaga. 

PE/IE Broadcast Section 

This is a chip-tr>common control device interface. It can be used by the device that serves as a 
<s controller to cwnmand VO, or test and diagnose the complete chip. 

siput Is word sequences feJBisr Instruction or data) thst a/9 available to subsets of PME*. Associated 
wtfh each word Is a code Indicating which PME* vriB use- ihe word. The Ilia BCl will use the word both to 
limit a PME*e access to flie interface and to assure that ell iequired PMEe receive data. TWs serves to 
adjust the BCl to the asynchronous PME oporeraona (Even when in SIMD mode PMEs are asynchronous 
so due lo I/O and Interrupt processing J Ihe median bm permits PMEs to be Tormed into groups which are 
controlled by Interleaved sets of commanci'dsta words received over the BCl. 

Besides delivering data to the PMEs, ihe SCI accepts request codes from tte PME combines from, 
and transmits Ihe Integrated request. This mechanism can be used In several ways. MMD processes can 
be fnmated In a group of processors that al! end with an output signal. The 'AMD* of signals triggers the 
os controller to initiate a new process. Applications, in many cases, will not be able to bad all required &W in 
PME memory. Encoded request » 3ie controller win be used to acquire a Saw overlay from perhaps the 
host* s storage system* 
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The controller uses a serial scan loop through many chips to acquire information on individual chips or 
PMEs. These loops Initially interface to th& BCl but can in the BCl be bridged to individual PMEs. 

Broadoast and Control Inter face 

9 

The BCl broadcast and control interface provided on each chip provides a paral&t Input interlace such 
that data or instructions can be sent to th» node. Incoming data is tagged with a subset identifier and the 
BCl includes the functions required to assure that all PMEs within the node, operating vaithin the 9tibset k are 
provided the data or instructions. The parallel interface of the SCI serves both as a pott to permit date to be 
w broadcast to all PMEs and as the instruction interface during SIMD operations. Satisfying both requirements 
plus extending those requirements so supporting subset operations is completely unique to this design 
approach. 

Our BCl parallel input interface permits data or instructions to be sent from a control element that ra 
external to the node. The BCl contains "group assignment* registers (see the grouping conoepfc in our 

js above appticafon erratled GROUPING OF SIMD PICKETS) which ere associated with each ot the PMEs. 
Incoming date words are tagged with a group Identifier and tire BCl Includes the functions required to 
assure that all PMEs wtfoin the node which are assigned to the dedicated group are provided the data or 
insfructions. The parallel interface ot the BCl serves as both a port to permit data to be broadcast to the 
PMEs during MIMD operations, and as ihe PME InstacUcfl/fomeOate operand Interface during SIMD 

20 operations. 

The 6CI also provides two serial interfaces. The Uty speed serial port vrifl provide eech PME with the 
capability to output a limited amount of status information. That data is intended to: 

1. signal our Array Director 610 when the PME, *,g. 500, has data that needs k> be read or that the PME 
has completed some operation. It passes a message to the external control element represented by the 

20 Array Director. 

2. provide activity status such that externa) iest and monitor element* can illustrate the status or fts 
entire system. 

The standard serial port permfes trie external control element means for selectively accessing a specific 
PME for monitor and control purposes. Daze passed over this interface can direct data from the BCl parallel 

so Interface to a particular PME register or can select data from a particular PME register and route it to ttie 
high speed serial port. These control points allow the external control element to monitor end control 
individual PMEs during Rrdtrat power up and diagnostic phases- It permits Array Director to input control data 
so as to direct Hie port ro particular PME and node internal registers and access points. These registers 
provtde paths such mat PME of a node can output data to fre Array Director, and these registers permit fh» 

as Array Director to input data to the unite during initial power up and diagnostic phases. Data input to access 
point can be used to control lest and diagnostic operations* to. perform single instructs step, stop on 
compare, break points, eta 

Node Topology 

Our modified hypercube topology connection is most useful for massively parallel systems, while other 
Jess powerful connections can be used with our basic PMEs* Within out initial embodiment of the VLSI chip 
are sight PMEs with ajty distributed PME interna! hardy/are connections. The Internal PME to PME chip 
configuration is a two rings of four PME&. with each PME also having one connection to a PME in the other 
ring. For the case of eight pubs m a VLSI chip this is a three dimensional binary hypercube. oowever our 
approach In general does not use hypercube organizations within the cNp. Each PME also provides for the 
escape of one bua in the tnllal embodiment the escaped buses form one ring are caSed +X +Y. + W and 
+Z> while those from the othei ring ere labeled similarly except - (mkms). 

The specific chip orgarweoSon is referred to as the node of tie array, and a node can be in a cluster of 

ao the array. The nodes are connected using +-X and +-Y into an array, to create a cluster. The 
dimensionality of fte anay is arbitrary, and In general greater man two which is the condition required for 
developing a binary hypercube. The clusters are then connected using +AN t + -2 into a array of clusters. 
Again, the dimensionality of the array Is arbitrary. The result is (he 4 -dimensional hypercube of nodes. The 
extension to a S-dtrnendiona! hypercube requires the usage of a 10 PME node and uses the additional two 

35 buses, say +-EI lo corned the 4-dimensional hypsrtube into a vector of hypercubss. We have tien shown 
the pattern of extension to either odd or oven radix hypercubes. This modeled topology limits the cluster to 
cluster wiring while supporting m advantages of the hypercube connection. 
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Our Ynroabiljfy and topology configuration for massively parallel machines has advantages in keeping 
me X and Y dimensions wHhm our cluster revel of packaging, and in distributing (ho w and Z bus 
QormediQrtS to aS the neighbor ing clusters- After impJemanrriKg the technique* described* the product w3l be 
wireable, and mantifecturabte while maintaining the inherent characteristics of the topology dofined, 

s The node consists of k ' n PMEs plus the Droadcast and Control Interface (BCl> section. Here *n" 
represents the number ot dimensions or rings, ?rhlcti characterize the modrted hypercube. while 'V 
represents the number ot rings that characterize the node. Although a node can contain U rings k is a 
characteristic of the concept that only two of those rings may provide escape buses, V and "k" is limbed 
in our preferred embodfrnent, because of tte physics* chip package to N«4 end k*2. Thrs limnetion is a 

70 physical ona, and different chips sets *3J allow other and increased dimensional/ in the array. In addition 
to being a part of the physical cbfr package, n Is our preferred embodiment to provide a grouping of PMEs 
thai totfttttnnect a set of rings m & modified hypercube. Eacn node will nave 8 PMEs with their PME 
architecture and abtlfty to perform processing and data router functions. As such, n is the Dimensionality of 
the modified hypercube (see following sectfon), a 4d modified hypercube's node element would be 3 

« PMEs while a 5d modified rtypsmibe's node would be 10 PMEs. For visuatizsaion of nodes which we can 
employ, refer to FIGURE 6> as well as FIGURES 9 and 10 for visualization ol Interconnections and see 
figure 11 for a block diagram of each node. FIGURES 16 and 16 elaborate on possible stterconnettions 
tor an APAP. 

It wtH be noted (hat the application entitled "METHOD FOR INTERCONNECTING AND SYSTEM OF 

20 I INTERCONNECTED PROCESSING ELEMENTS 1 ' Oi GO-iirventQr David B. Rohe, «ed in the United St$te9 
Parent and Trademark office on May 13, 1981, under U3SN 0778&8,868. described the modmed hypercube 
criteria which can preferably be used In connection with our APAP MMR 

That application is incorporated by reference and describes a method of Interconnecting processing 
elements in such a way that the number of connections per element con be balanced against the network 

£9 diameter (wore; owe path length*. This Is done by creating a topology that maintains many of the well 
known and desirable topotogeal properties of hypercubes while improving rrs flexibility by enumerating the 
nodes ol the network in number systems whose base can be varied. When using a base 2 number system 
this method creates fre hypercube sopoiooy* The invention has fewer ^connections than a hyperciibe. 
uniform connections and preserves the properties ot a hypercube. These properties include: i) large 

30 number of alternate pa(hs> 2) very high aggregate bandwidth, and 3) well understood and existing meilwds 
lhat can be used to map other common problem topcJogiea with the topology of the network The result is a 
genera! bred non-binary hypercube with less density. It mO be understood that with the preference ws have 
given to the rncdffced hypercube approach. In some applications a conventional hypercube can be utilized. 
In connecting nodes, other approaches to a topology could be used; however, the ones we describe herein 

as are believed to be new and an advance, and we prefer the ones we describe. 

The Interconnect methods for the motffled hyperetbe topology for tnieronnecting a plurality ol 
nodes in a network ot PMEs; 

1, defines a sets of integers el, e2, eS. ~, such the product of all elements equals toe number of PMEs 
an the network called M while the product of all elements in the set excepting el end e2 is the number 

<* ot nodes caned N, and the number of elements in the set called m deiiftea the dimensionality of the 
network n by the retaticcwrwp a = n>2. 

2. odd-esoes a PME located by a set ot indexes e1, a2, ... am. where each Index Is the PMEs position in 
the equivalent level ot expansion such that the Index al is in the range ot zero to eM tor i equal to 1. 2. 
to nr. by ttie formula 

Ma<m)*e(nvl) + a (m»2»ef>n-1) ^sWe(1»+a(1) 

wttere the notation a{i) has the normal meaning of fbe me ith element in the list of elements called a, or 
squrvatantfy for e. 

so 3. connects two PMEs {wUh addresses f and g) rf and only if eKher ol the following two conditions hold: 

a. the stfeger part ot r/(ei ' e2) equals if * Integer part of s/(ei ' e2) and: 

1 . the remainder part of rfel drffere from the remainder pan o* s/et by 1 or, 

2. the remainder part of rfe2 differs from the remainder part of s/e2 by 1 or e2-i. 

b, me remainder part of r/ai defers frorri the remainder part of are! for I m the range 3. 4. m and the 
m remainder part ol r/et equals the remainder part of s-fe£ which equals i minus three, and the 

remainder pert of r7e2 differs from the remainder part of s/e2 by e2 minus one. 
As a result the computing system nodes will form a non-binary hypercube, wrih the potenaal ior being 
diherent racfix in each dimension. The node is defined as an array of PMEs which supports Zn ports such 
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that the ports provided by nodes match the dimensicns&ty requirements of the modified hypercube. If the 
set or integers S3. e4. ... em. wnfch define the speerfic extent of aacn dimension or a particular modified 
hypeicube are e» taken as equ^l, say b, and if el and 92 ei© taker* a 1, then the previous formulas for 
addressability and corjnoctions reduce to: 

Z addressing a PME as ambers representing me base b numbering system 

3. connecting two computing elements (f and g} if and only ff the address of f differs from tha address of 
g in exactly one base b digit using the rule that 0 and b-i are separated by 1- 

4. the number of oonnecHocre supported by each PME is 2*n 

» Which t& exactly aa da scribed in the base application, with (ha number of communication buses spanning 
noo-edfacerc pmes chosen aa *ero. 

Intra-htodo PME Neroonnecticre: 

;s PMEs ore configwod within the node as a 2 by n array- Each PME ta interconnected with fe three 
neighbors (edges wrap only in the second dimension) using a set of input/output ports, thus; providing fiil 
duplex cornrnurncatlon capability between PMEs. Each PMEs external input and output port is connected to 
node hO pins. Input and output ports may be connected to share pine for haff-duptex communication or to 
separate pins for ful Wupte* capability^ The imerconnecUona tor a id modified hypercube nods are shown 

so in figures? $ and 1 0. (Mote that where n i$ even fro node cam be considered to b* a 2 by 2 by n/2 array.) 
FIGURE 9 rJlu&traies the the eight processing etemenxs 500, 510, 520, 63ft, 540, 550, 500, 570 wftMh 
the node. The PMEs am coenected in a binary hyperoute communication network. This etnar? hypercube 
displays three intra connections betwfcan PMEs (501, 51 1 521, 531, 541. 551. 561, 571, 590, 591, 592, 593), 
Communication between the PME is contacted by in and out registers under control of the processing 

£$ element This diagram shows the various paths that can be taken to escape K> out any of the eight 
direction*, 525. 565. +-x 515. 555. 505, 545, +-z 535, 575. The communfcasion can be 
accomplished without storing the data Into memory if desired. 

it may be noted that vehBe a network switch cWp could be employed to conned various cards each 
having our chip wfth other chips of me system, it can and shortd dasvabty pa eliminated. Our toier PME 

so network that we describe as (he "4d torus" is the mechanism used for Inter PME^ommunication. A PME 
can reach any other PME hi fre array on mis interface. (PMEs in between may be Store^PoAvaid or Circuit 
Switched) 

Chip Relationships for IntarDonneciiona 

79 

We have discussed ihe chip, and FIGURE tt shows a block (lep/m 0 t the PME PrccessotfMemory 
chip. The chip colters of m foJIov/ing efemettis each of which will be descrtoed in later paragraphs; 

1. 3 PMEs each consisting of a Id bit programmable processor and 32K words of memory <6*K bytes), 

2. Broadcast Interface (BO) to permit a controller to operate all or subsets of the PMEs end to 
<o eccumulate PME request, 

3. Interconnection Levels 

a. Eacft PME supports tour 8 bit wide tnter-PME communica&c* paths. These connect to 3 
neighboring PMEs on the chip and 1 off chip PME, 

b. Braadr_asMoPME hueinrj. which makes data or instructions available. 

*s c. Service Request lines that permit any pme to send a code to the controller. The bci combines tna 
requests and forwards & summery. 

d. Serial Service loepe permit (he oontrellsr to read aB detail about fte functional blocks. This level of 
interconnection extends from the Bd to all PMEs (FIGURE 11 fa eaee< of presentation omits this 
dels*).) 

30 

toterconnecaon and Routna. 

The MPP w?H be Implemented by repficelcn of tfie PME. The degree of repBcatton does not affect the 
interconnection and rowing schemes used, FIGURE 6 provided an overview of the network Interconnection 
sa schsme. The chip contains 6 PME3 with interconnections to their immediate neighbors. This interconnection 
pattern results in #e three dimensional cube structure shown in FIGURE 10. Each of the processors within 
the cube has a dedicated MdlrecSonei byte per* to the cNp + & pine; we refer to the set of G PMEs as a node. 
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An n by n array of nodes is a duster, Sirnpte bridging between the * and - x ports and the + and - y 
pone provwe the caueter node intercom neefcona Hero the our preferred chip or nods has 8 PMEs 4 each of 
which manages a single externa! port This debutes tne network control function and eliminates a possible 
bottleneck for pons. Bridging the outer edges mates the cluster into a logical torus. We have considered 
5 clusters with n = 4 and n * 8 and befceve that n = B Is the better solution tor commercfil applications while 
n»4 te better fo? miStary conduction cooled applications. Our concept doss not hnpoee art unchangeable 
cluster size. On the corarary, we anticipate some applications using variations. 

An array of clusters lesuits In the 4 dimensional torus 0i hypercute structure illustrated In FIGURE 10. 
Bridging between the * and - w ports and + end - z ports provides the 4d ioms interconnections- This 
70 nssdte in each node within a cluster connected to an equivalent node in all adjacent clusters. (This provides 
©4 ports between two adjacent dusters rather then the 3 ports that would result from lergef dusters.) As 
\v»m the cluster size, the scheme does not imply a particular size array- We have considered 2x1 arrays 
desirable tor workstations and MIL applications and 4x4. 4x8 and 8x8 arrays for mainJrama appfications. 

Developing an arcs? of 4d foruses Is beyond the gate, pie, and connector limitations of our current 
>g preferred chip. However, *ai limitation disappears with our alternative on-chip optical dnver/raceiver is 
employed. In ihte ombodimeni our network could use an optical path per PME; wBh 12 rather than 8 PMEs 
pet chip the aney of 4d toruses with muffl-Tflop (Teraftap) performance end It seems to be economical? 
feasible to make such machines available for the workstaSon environment Remember that such alternative 
machines wrfil use the application programs developed for our current preferred embodiment. 

a* 

46 cluster Organisation 

For constructing a 4d modified hypsreubs 360, as illustrated in FIGURES 8 and 10, nodes supporting 8 
external ports 31$ are required. Consider Ins external ports to be labeled as +X, *Y. +2, +W, ~K -Y, -2, 

29 -W. Then using nodes, a ring can be construed by connecting the to 0< pons. Again m 2 such 
rings car> be rm^nconriecied into a ring of rings by irrterconnectrng the matching +Yto -Y ports. This level 
of structure will be called a cluster 320. With mi =m 2 -3 it provktee for 510 PMEe and such a cluster will 
be a building block for several stee systems (330, 340, 350). as IBusfcated with m=8 In RQURE 6. 

30 jj Array Organization 

For building large finegrained systems, sets of m3 dusters are interconnected in rows using the * Z to 
•Z posts. The m< rows are then interconnected using the * W to -W ports. For mi «..jtu » 8 this results In 
system with 32763 or 8*" PMEs- The organization does not require that every dimension be equal y 

35 populated as shown in FIGURE 6 {large tine-grained parallel processor 370). In the case of the finegrained 
9ma§ processor, only a duster might be used with the unused Z and W ports being interconnected on the 
card. This technique saves caid connector pins and makes possible the application of this scalable 
processor to workstations 340, 350 and avwrfe applications 330, both of which are connector pin limited. 
Connecting */• ports together In the Z and YV pairs leads to a workstation organization that permits debug. 

45 test and large machine software development. 

Again, much smaller scale versbns of (he structure can be developed by generating the structure with a 
value smefler than m = a. This will permit construction of single card processors compatible with the 
raqujremenss for acoetarstors m the PS/2 or RISC System 6000 worfcatatton 350, 

<8 l/Q Performance 

VO performance Includes overhead to setup transfers and acfcai burst rate data movemani Setup 
overhead depends upon application function K> complexity and network contention. For example* an 
application can program circuit switched traffic with buffering to resolve conflicts or it can have all PMEs 
so transmit left and synchronize. In fte first case. I/O is a major taste and detailed analysis wouM be used to 
size it We estimate that simple case setup overhead is 20 to 30 clock cyctea or B to 1,2 u-sec. 

Burst rale VO is the maximum rate a PWG can transfer dara (with either an on or off chip neighbor,} 
Memory access Dnrits set the data rate at 140 nsec per byte, corresponding to 7.14 Mbyte/s. This 
performance includes buffer address and count processing ptos data reaowrite- It uces ssven 40ns cycles 
ss per IS bit word transferred. 

This burst rate performance coircsponcte to a duster having a maximum potential transfer rate of 3*5 
Gbytea's- K also means that a set of eight nodes along a row or column of ihe duster will achieve 57 
Mbyte/s burst data rata using one set of their 8 avaifebte ports. This number is significant because I/O with 
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ths external work* wffl be done by logically 'unripping 1 an edge of the wrapped duster and attaching it to 
the externa! system bus. 

tnter-PME Routing Protocol 

9 

The SIMD/MIMD PME comprises taferproceasor communication to trie external VO faciimes, broadcast 
corwroi Interfaces, and switching femures which aflow bath SlbtOrtrilMD operation whhfci the same PME, 
Embedded In the PME Is me fuBy olsWbuled programmable I/O router for processor communication end 
data transfers between PMEe. 
70 The PMEs have fully distributed mterprooeEscr communication hardware to on-chip PMEs well as to 
the external VO facilities which connect to the interconnected pmes tn the modified hypercube oonfigura- 
tfort This hart*** is complemented with the flexible programmabilRy of the PME » control the to activity 
via software. The progmrnrnable I/O router functions provide for generating data packets and packet 
addresses. With this Information the PME can send the information thru the network of PMEs in a directed 
/s rralbod or out multiple psths determined by any fault tolerance requirements. 

Distributed (auk tolerance algorithms or program dgortftms can Lake advantage of lha prcnraiTimafoiSty 
along wfih the supported circuit switched modes of the PME. Thfe performance combinafiond mode 
enables everything from ofHine PMEs or optimal path data tfructuree to be acoom pished via fte 
program rnaWe I/O router. 

4> Our study of sppneafons revels that it is sometimes meet efficient to send bare data between pme*. 
At other times applications require date and routing information. Further, it is sometimes possible to pten 
communications so (hat neiwodc oonficts cannot occur: other appiicetions oner the potential tor deadlock* 
unless mecrarfems for buffering messages at intermediate node* are provided, Two examples ttustraie me 
extremes. In the relaxation phase of a PDE solution, each grid point cca be etiocated lo a node. The inner 

£e loop process of acquiring data from a neighbor can easily be synchronized over en nodes. Alternatively, 
imag* treitdontiatton* use local date pa*™***** to determine communrcation target or source identifiers. 
Thie results in data moves tnrouoh multiple PMEs, and each PME becomes involved in the routing task lor 
each pecket Preplanning such traffic is generally not possible- 

To enable the network to be efficient tor all types ot transfer requirement, sve partraon, between me 

30 hVW and 8/W« the responetbiliy tor data routing between PMEa &/YV does most of the task sentencing 
function. We added special features to the hardware (HrW) to do the inner loop transfers and minimize 
software (8/W) overhead on the outer loops. 

I/O programs at dedicated interrupt levels manage the network. For most applicator*, a PME dedicatee 
tour antewupt levels to leceiving data from the four neighbors. We open a buffer aj each tevel by loading 

35 registers at the level, and executing the IN (ft uses buffer address and transfer count but does not await 
data receipt) and RETURN instruction pair. Hardware then accepts words from ihe particular input bus end 
stores them to She buffer. The buffet full condition will men generate tie Interrupt and restore fre program 
counter to ihe tnsiruolion after the RETURN. This approach to interrupt levels permits I/O programs to be 
written that do not need to test what ceased the interrupt Programs reed data, return, end then conSnue 
directly into processing me cteta they read. Transfer overhead is minimized as meet eKuaSon* require K9e 
or no register saving. Where an application uses synchronization on I/O, as in (he POE example, tfien 
programs can be used to provide that capability. 

Writ* operations can be started m several ways. For the PDE example, at the point where a result la to 
be sent to a neighbor, the appEcailon level program executes a write call. The call provides buffer location , 

<$ word count and target address. The write subroutine Includes the reoteter loade end OUT instructions 
needed to initiate) the H/W and return to the application. H'W does the actual byte by byte data transfer. 
More complicated output requirements w>!) use an output service function at tie highest Interrupt level. Both 
application and interrupt level tasks access that service via a soft interrupt. 

8e»Ing up circuit swaensd paths builds on these simple read and write operations. We start wSh sU 

ssa PMEs having open buffers steed to accept packet headers but not the data. A PME needing to send data 
initiates the transfer by sending an address/data block to a r*ightt»ing PME whose address better tranches 
the target b the neighboring PME address information wttl be stored; due to buffer full an interrupt wit! 
occur. The mterrupt tests the target address end will either extend the buffer to accept the data or write 
the target address to an output pott aid set the ctl register for transparent dasa movement. {This aiiowe 

so the PME to overlap its application executions with the circuit switched bridging operation.) The CTL register 
goes to busy state and remains transparent unttf reset by the presence of a signal at end of data stream or 
abnormetfy by PME programming. Any number & variations on these themes can be in^lemenied. 
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System I/O and Array Director 

FIGURE 12 $ho>*9 an Array Director In the preferred embodiment, which may perform the functions 0* 
the controller of FIGURE 13 which describes the system bus to array oorsiectioro, FIGURE 13 b composed 

s of two parts, (a) the bus to/from a cluster, and part (b) toe communication of Information on the bus to/from 
a PME. Loading or unioad&ng the Bray b done by connecting Hie edges of clusters u> tl>e system bus. 
Multiple sysrem buses can be supported wift multiple dusters. Each duster supports 50 to 57 Mbyte's 
bandwidth. Loadbng or unloading the parallel array requires moving data between all o* a subset Of the 
PMEe and standard buses (ie MicroChannel, VME-bue, FutureBue, etc). Those buses, part of the host 

;o processor or array controSer, are assumed to be rigidly specified. The PME Array therefore must be 
adapted » *e buses. The pme Array can be matched to the berwjwkfch of any bus by interleaf bus data 
onto n PMEs, with n picked to permit PME s bom I/O and processing lime. FIGURE Id shows how we might 
connect the system buses to the PMEs as two edges of a cluster. Such an approach would permit 114 
Mbyte* to be supported. It also permits data to be loaded at half the peak rase to two edges Simula- 

55 neousry. Although this reduces ths bandwidth to 57 Mbyte's/cluster, ri has the advantage of providing 
orthogonal data movement within (he array and ability to pass data between two buses. (We use those 
advantages to provide test transpose and matrbc mutipJy operation.) 

As shown fcn part (a) of FIGURE IS, the bus "dots to aB pains on too edges at the cluster; end, the 
controller generates a gate signal to each path In the required Intertesva timing. If required to connect to a 

so system bite wnh g»eeie» then $7 Mbyfe/e, the deia wffl be interleaved over multiple clusters. For example, in 
a system requiring 200 Mbyte/5 system buses, groups of Z or 4 clusters will be used. A large MPP has the 
capacity to attach 10 or 94 such buses to its xy network paths. By using the w and z paths In addition to the 
x and y paths, that number could be doubted. 

LIRE 13 part (b) shows how the data routes to individual PMEs. The FIGURE shows one particular 

2$ w>,y or 2 path that can be operated at 7.13 Mbyte* In burst mode- If the dete on the system bus occurred 
prt bursts, snd if the PME memory could contain ihe complete burst, then only one PME would be mqurred. 
We designed the PME I/O structure to require neither of these conditions. Data can be gated into the 
PME*0 at the fell rate urrrj buffer full occurs. At that instant PME*0 will change to transparent and PMBci 
wai te0in accepting 6m. Within PMExO processing of the input date buffer cm begin. PMEs mat have 

a> taken daia and processed It are limited because they cannot transmit ths results while in ihe transparent 
mode. The design resolves thio by switching the dafa stream to the opposite end of the pafr ai intervals. 
FIGURE 13(b) shows that under S*W control one might dedicate PMExO through PMExd to acceding data 
while PMEk12 through PMEx15 unload respite and vfea-verea. The controller counts words and adds end of 
block signals to the data stream, causing the switch in direction. One count applies to all paths supported 

33 by the controller so controller workload is reasonable. 

SYSTEMS FOR ALTERNATIVE COMPUTERS 

FIGURE is Illustrates a system block Diagram tor a host attached large system with a singe application 

& processor interface (APT). This iflusiration may afeo be viewed wnft the understanding that our invention may 
be am ployed in stand alone system which use multiple application processor interfaces (not shown) Thfc 
configuration win support DASEVCmhplcs on all or many clusters. Workstation ecceierators can eliminate the 
host, eP0ics3on processor Interface (API) and cluster synchronizer (OS) Illustrated by emulation. The CS 
not always required. It wiS depend on Use type ot processing that Is being performed, as well as the 

4$ physics* drive or powei provided for e particular application which uses our invention. An epoicatton this is 
doing mostly MIMD processing wil not place as high a workload demand on the con totter, so here the 
control bus can see very stow pufoe rise times. Conversely k system doing mossy asynchronous A-SIMD 
operepons with many independent groupings may reQirire faster control busing, in this case, a ciisier 
Eynchroncrer will be dearaDte. 

so The system block dagram of FIGURE 18 Illustrates that a system might consist ol host array conferoSer 
and PME array. The PME array © a set of clusters supported by a set ot cluster controllers (CC), Although 
a CO is shown for each cluster that relationship is not ssrictiy required. The actual ratio of clusters to OCs is 
fiexlbfe The CO provides redrlve to, and accumulation from the 64 BCIsfclusiers. Therefore, physical 
parameters can be used establish the maximum ratio. Additionally, the CC win provide for controlling 

ss multiple independent subsets of the PME array: lhat service might also become a gating requirement A 
Study can be made to determine these requirements for any particular application of our inventor*, two 
versions of the CC will bs used. A cluster dial is to be connected to a system bus require* the CC providing 
interleave controls (see System I/O and FIGURE IB> and hi-etste drivers. A more simple version lhal oro&s 
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ths tri-Etete busing features can also be employed. In the case of targe systems, a second stage of red rive 
and accumulaiion to added. This level is the duster synchronizer (OS). Hie eat of CCe plus CS and the 
Aerification Processor Interlace (API) rna^ up the Airay Controller. Only {he API i* a pro^r^mmeWe 

Several variations of this system synthesis scheme wHI be used These rest* in different hardware 
3 configurations for various application^ but Ihey do not have a major Impact on the supporting software* 

For a workstation accelerator, the ctasfer conirotters will be attached directly to the workstation system 
bus; me function of me API wtl be performed by the workstation. In toe case of a RI8G8000, ihs system 
bus is * Micro Channel and the CC units can plug cfrecfly into the slots wftnin the workstation. This 
configuration places the K> devices (QASO, SCSI and dtepfery interfaces) on the same bus that 
ro ioadsainioada the array. Ac such the parallel array can be used Cor VO inlensrve tasks like real time Image 
generation or processing* For workstations using other bus systems {VME-ous* FufcireBus. a gateway 
Interface *vM be used. Such modules are readily avsaabte in tie commercial marketplace. Note mat in ftes® 
minimal scale systems a single CC can be shared between a c&enrvined number of clusters, and neither a 
OS nor an api is needed 

& A MIL avionics application might be simiier in size to a workasrfon, but it needs d&feretit interfering. 
Consider what may become the normal mffllary situation. An existing platform must be enhanced with 
additional processing capability, but funding prohibits a complete processing system redesign. For this we 
would attach to the APAP array a smart memory coprocessor, in this case, a special appication program 
Interface API thai appears lo the hosi as memory will be provided. Data addressed lo the host's memory 

so waf then be moved to the array vie CC($). Subsequent writes to memory ceo be detected end interpreted as 
commands by the API eo thai the accelerator appears to be a memory mapped coprocessor. 

Large systems can be developed as either host attached or as stand alone ccnflgurations. For a host 
attached system, ihe conf^guraaon shown in FIGURE 18 is useful. The host will be responsible for I/O, and 
the API would serve as a dispatched task manager. HotvBver, a large stand slone system is also possibte in 

so special situations. For example, a database search system might eliminate the host, attach DA$D to the 
MicroChannel of every cluster and use multiple APIs as bus masters slaved to the PMEs. 

Zipper Airay Interlace wfri External I/O 

so Our zipper provides a fast MO connection scheme and is accomplished by placing a switch between two 
nodes of the array. This switch wll allow for the parallel communication into and out of the array. The fast 
VO witi be implemented along one edge of the array rings and acts like a large zipper into the X, V. W„ Z 
rings. The name "zipper connection" is given to the fast VO. Allowing date to be transferred Into and out of 
the network while only adding switch delays to transfer the data between processors is a unsqus loading 

35 technique. The switching scheme does net disrupt the ring topology created by the X, Y, W, Z buses and 
special support hardware slows the atpper operation to occur while the PE is processing or routing data. 

Ths ability to bring data into and out of a massively parallel system rapidly is en important 
enhancement to the performance of the overall sysiem,We oelieve lhal the way we implement our fest MO 
without reducing the number of processors or dimension of the array network is unique in the field of 

«9 massively parallel ertvironments, 

The modified hypercube arrangement can be extended k> permit a topology which comprises rings 
within rings. To support tfie interface to the external NO any or all of me rings can be logically broker). The 
two ends of the broken ring can than be connected to external I/O buses. Breaking the rings is a logical 
operation so as to permit regular inier*PME communicairon at certain fime intervals white permitting HO at 

*s other 3me intervals* This process of breaking a level of rings within the modified hype* cube effectively 
'unzips* rings for UO purposes. The fast WO "zipper " provides a separate interface info tf» array. This zipper 
may exist on t to n edges of the modified rrypercubs and could supper* either paraiel input into muffipie 
dimensions of the array or broadcast to multiple dimensions of tte stray. Further data transfers into or out 
of the array could alternate between the two nodes drecrty attached to the zipper. This I/O approach is 

so unique and it permits developing different zipper sizes to satisfy particular application requirements. For 
example, in the particular configuration shown m FIGURE 6, called me large fine-grained processor 36u, the 
zipper for the Z and W buses will be dotted onto ihe MCA bus. This approach optimizes the matrix 
frenspositlon tfcne-, satisfying a particular application requirement f or (he processor. For e more detailed 
explanation of the "zipper" structure, reference may be had to the APAP \fO ZIPPER application tiled Hied 

55 concurrently herewith. The zipper is here ilualrated by Figure 14. 

Depending en the configuration and the need of the program to roll date and program into and out of 
ths individual processing elements, me size ol the zipper can be varied. The actual speed of the K> zipper 
is apprownalely the number of rings attached limes Ihe PME bus width, times the PME clock rate all 
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divided by 2. (The drvision permas lb© receiving PME time to move data onward. Sine* H csn send It to any 
or n places I/O contention to completely absorbed over the Array.) wftfi existing technology. Ie_< 5 mb/ssc 
PME transfer rate, 64 ring? on the- zipper, and Interleaved to two nodes transfers, 320UB/$ec Array tender 
raise are possible- (See ths typical zipper corrngureoon in FIGURE 14). FIGURE 14 illustrate* the feet I/O or 
s the so-called 'zipper connection* 700, 710 which exists as a separate Interface Into trie array. This zipper 
may exist on one 700 oi two edges 700. 710 ot the hypercube network by dotting onto the broadcast bus 
720, 730, 740, 750, al multiple nodas In the array 751, 762, 753, 754 and In mutfpto direcrjons 770, 760, 
700, 755, 767. 

Today 1 * MCA bus supports 60 to 160 MB per second buret transfer rate and therefore la a good match 
to for a eingfo zipper in simple or non-intorleeved mode. The actual transfer rata given channel overhead and 
efficiency is ocmetrring less than that Fix systems that have even more demanding VO requirements, 
multiple zippers and MCA buses can t» utilized These techniques are s*en to be important to processors 
that *ouJd support a large externa! storage associated wilh nodes or clusters, as might be characteristic of 
database machines. Such I/O growt> capability Is completely unique to fr»9 machine and has not previously 
»3 been incorporated in either massively parallel, conventional single processor, cr coarse-grained parallel 
machines* 

Array director Archrtecturo 

a? Oi* massively pareflei system is made up of nodal building Nocks of muieproce$$oi nodes, dusters of 
nodes, and arrays of PMEs already packaged in clusters. For control of these peckaged systems we 
provide a system array director which with the hardware controllers performs the overall Processing 
Memory Entrant (PME) Array Controller functions in the maatfvery paraBel processing emironrnant The 
Director comprises ot three functional areas, the Application Interface, the Cluster Synchroniser, and 

a? normafly a fluster Controller. The Array Director vsfil hav? the overall control ot the PME array, using the 
broadcast bus and our zipper connection to steer data and commands to all or the PMEs. The Array 
Director functions as a software system Irrteractlng with, the hardware to perform (he role as (he shell of the 
operating system. The Array Director In performing thte role receives commands from the application 
irtiarfcee and issuing the appropriate array instructions ens hardware* seQuertees to accomplish the 

so designated task. Hie Array Director's main function Is to continuously Teed the Instructions to the PMEs and 
route data in optimal sequences in keep the traffic at a maximum and collisions to a minimum. 

The APAP computer system shovm in FtOOf£ 6 is lltusfrsrted in more detail in the oisoram of FK2URE 
12 which Illustrate* the Array Director which can function as a controSer, or array controller, m iiustrated In 
FIGURE 13 and FIGURES 18 and 19. This Array Director 610 illustrated in FIGURE 12 i$ shown in the 

35 preferred embodiment of an APAP in a typical configuration of n identical array dusters 605, 670, 6SO, 630. 
with an array director $10 for the clusters of 5i2 PME$> end on application processor Interface 030 for the 
application processor or processors 600. Hie synchronizer 650 provides lite needed sequences to the array 
or duster controller 040 and together (hoy make up ahe "Array Director" BlO, The applcation processor 
interface 630 wHI provide the support for the host processor 300 or processors and test/debug workstations. 

& For APAP unsa attached to one or more hosts, m Array Director serves as the interface between the user 
and $\& array of PWEs. For APAPe functioning as stand alone parallel processing machines, fte Array 
Director becomes the host unit and accordingly becomes HvoJved kn unit VO activities. 

The Array Director will consist of the tolkwtng four functional areas: (see to functional block oiarjram in 

FIGURE 1 2) 

<*5 1 , Application Processor mterfaoo (API) eoo, 

2, Ouster Synchroniser (CS) 650 (0x8 array of cluster*!, 

3, Oueter Controller {CC> 840 (8 x 1 array of nodes). 

4, Fast lr'0 (zipper Connection) 520. 

so ThB Application Processor Interlace (API) 630: 

When operating in attached modes, one API win be used for each hose Thai API writ monnor the 
incoming data stream to determine what ere instructions to the Array dusters 605. 870. 600, 680 and whet 
are data for the Fast *0 feipperi 020- When in standalone mode, m API serves as the primary user 
as program host 

To support these various requirements, the APIs contain the only processors within 3w Array Director, 
plus ihe dedicated storage tor the API program and command* Inductions received from trie host can call 
tor execution of API subroutines, loading of API memory with additional fonefcons. or for loading of CC and 
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PME mernory with naw S/W. As cteBoribed in die Sn?V overview section, these various type requests can be 
restricted to tubes* or users via the initial programs loaded Into ihe API. Thus, the operating program 
loaded «ai determine the type of support provided which can be tailored to march the performance 
capability of the API This further permits the APAP to bo adjusted to ft© needs of multiple users requiring 
s managed and wel tested services, or to the Individual user wishing to obtain peak performance on a 
particular application. 

The API also provides for managing the paft to and from the vO zipper. Data received from the host 
system In attached modes, or from devices in standalone modes Is forwarded to the Array. Prior to initiating 
this type of operation the PMEs wKhm th8 Array wnich will be managing the K> are Initiated. PMEe 

*> operating in MIME> mode can utilize ttie fast interrupt capability- 2nd either standard SAV or special functions 
for this transfer while those operating in SIMD modes would have to be provided detailed control 
ins&uciiona. Data being sent from the VO zipper requires somewhat the opposite conditioning. PMEe 
operating, in MIMD modes must signal (he API via the high speed serial infer face and await a response from 
the API, whin PMEs In SIMD modes are already In synchronization) with the API and can therefore 

is immorJately output data. The ability to system switch between modes provides a unique ability to adjust tie 
program to the application. 

Cluster Synchronizer (C3) 650 

20 The OB 650 provides the bridge between the API 630 and CC 640- It stores API 630 output In FIFO 
stacks and monitors the status being returned from ihe CD 650 (both parallel input acknowledges and high 
speed serial bus data) to provide trie CC t tn timely fashion, with the desired routines or operations that need 
to be started. The CS provides the capability to support different CCs and different PMEs within clusters so 
as io permit dividing the array into subsets. This is done by partitioning the array and then commanding the 

as involved duster contrgBero fe> s^l-^ctlv^ly forward the deseed operation. The primary functon of the 
synchronizer it to keep all clusters operating and organized such that overhead time ie minimized or buried 
under the covers of PME execution time We have described how ihe use of the duster synchroniier in A- 
SIMD corrf durations is especially dessable. 

so Cluster Oontoter (CC) 640 

The CC 640 interfaces to tfw node Broadcast and Control Interface (BCI) 605 far the set of nodes in an 
array duster 655. (For a 4ti modified hypercube with 8 nodes per rtng that means the CC 340 la attached to 
64 BCfcs 605 in an 8 by 6 array of nodes and is controlling r>i2 PMEs. Sixty~«ouv such clusters, afeo In a 8 

as by 6 array, lead to the tvD up system wBh 32788 PMEe,) The CC 640 wiH eend commands and data 
suppAed by t>e C$ 660 to the BCI parallel port and return the acknowledgement data to U*e C$ 660 when 
operating in MIMO modes, in SIMD mode the interface operates synchronously, and step by step 
acJmowfedgments are not required. The CC 640 also manages and monitors the high speed serial port to 
determine when PMEs within the nodes are requesting services. $uch requests are passed upward to the 

40 CS 650 while the raw data from m high speed serial internee i* made available to the etetue oleptey 
interface. The CC 640 provides the C3 650 with an interface to specific nodes within me cluster via the 
standard speed serial Interface. 

In SIMD mode the CC win be directed to send Instructions or addressee to all the PMEs over the 
broadcast bus. The CC can despatch 10 b& instruction to ail PMEs. every 40 nanoseconds whan in SIMD 

4$ mode. By oroaacasting groups or easve instructions to tne pme, the emulated instructor set to formed. 

When in MiNto mode the CC wait for tie endop signal before issuing new instructions io the PMEs, 
The concept of the bliMD mode is to build airings of micro-routines with nafiue inetajctlone resident In the 
PME. These strings can be grouped together to form the emulated Instructions, end these emulated 
Instruction can bs combined to produce service'canned routines or Hbrary twtcdons. 

so When In SlMDrtJDMD (SJMIMD) mode, the CC will issue instruction as If In SIMD mode and check for 
endop signals torn certain PMEa The PMEs that are in MIMD wdi not respond to tl>e broadcast instructions 
and will continue wftti there designated operation. The unique sratus indicators wJU notp ihe CC to manage 
this operation and determine when and to whom to present Ihe sequential instructions. 

55 Operational goftware Levels 

This applcation overviews the operational software $/W levels to provide further explanation of the 
services performed by various hardware HAV components. 
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Computer systems generally used have an operating system. Operating system kermis which arc 
relstiraiy compete must ho provided (n most massive MffdD machines, whore workstation class CPU diipe 
run karris such as Mach. The operating system k^rrvM ^ippor^ message passing or memory coherency. 
Other massively parallel systems beeed upon SIMD models have almost no intelligence in the array. There 
5 are no 'program counters'' out In the array, and thus no programs to execute locally. All instructions are 
broadcast. 

In the systems we have* provided with our PME as the basts for cluster arrays, there is not need for an 
operating system at each chip, a node- We provide a library of key function* for computation and/or 
communication wtthln each P6 (PME) thst can be tnvotisd at e high level SiMD-lfee instructions are 

jo broadcast to the anay to sat each of a selected eel of PMEs. These PMEc can then perform in {ell MIMD 
mode one ot more of these library routines. In adcfflon baste irnerupt handler and communications routines 
are resident in each PME £l)owir*Q, the PME ro handte comrmink^lon on a dynamic basis. Unlike existing 
ft/HMD machines* ihe APAP structure need no! include an entire program in PME memory. Instead all of Ehal 
code, which Is essentially serial, is the cluster convener. Thus such code, 90% by opaoe and 10% by time 

»a (typically) cen be broadcast in a SIMD feshb to an array of PMEs. Only the truly parsflel inner loops are 
distributed to Uie PMEs si a dynamic faslson. These are then initiated Into MIMD mode fust as other 
"library" routines are. This enables use of program models which are Single Program Muriipie data to be 
used whore the same program is to acted In each PME node, with embedded synchronization code, and 
executed at the local PME, Design parameters affect bandwidth available on different links, and the system 

20 paths are progremrr.£tically censurable, allowing high b*r?<lw*h links on a target network, and allowing 
dynamic parti Son of off chip lite PME-to-PME links to provide more bandwidth on specific paihs as meets 
the needs of a particular application. The links leaving a chip mate directly *tth each other, without the need 
for external logic There are sufficient links and there is no predesigned constraint as to which other Snts 
they can attach to, so thai the system can have & diversity of interconnection topologies, wift muting 

25 performed dynamically end progiemmetfc&lly. 

The system allows usag& of existing compilers and parsers to create an executable* paranal program 
which could run on a host or workstation based configurator Sequential source code for a Single Program 
Multiple Data system would pass through program analysis, tar examination of dependency, data and 
controls, enabling extension ot prcoram source fo include call graphs, dependency tabids, altos**, usage 

30 tables and the like, 

Therafter* program transformation r;ou)d occur whereby a rnotffied version of the program would be 
created that extends the degrees of parallelism by combining sequences or recognizing patterns to 
generate exp&cit compiler dlrecthres. A next step would be a data a? location and partitioning step, with 
message generation, which would analyze data usage patterned allocate so that efements to be combined 
would share common indexing, addressing pattern, and thess would provide embedded program compiler 
directives and calls to communication services. At this point the program would pass to a level ps/Utkming 
step, A level partitioning step would separate ihe program into portions for execution m ARRAY, hi ARRAY 
CONTROLLER (array director or cluster controller), and H08T. Array portions would be interleaved In 
sections w&h any required message passing synchroniza^on functions. At this point level processing could 
proceed. Host sources would pass to a i$vei compter (FORTRAN) tor assembly completion. Corwrefter 
sources would pass to a rntcroprooessor controller compiler, and items thai would be needed by a single 
PME and not available in a library caS wottid pass to a parser (FORTRAN OR C) to an intermediate level 
language representaston wwch would generate optimized PME code and Array Controller code. PME code 
would be created at PME machine teveL and would include library extensions, which would pass on load 

* into & pme memory. During execution a pme parages program in the SPMD process of execution could call 
upon already coded assembly service functions from a runSime ftbrary kernel. 

Sines the APAP can function as either an attached unit that is closely or loosely coupled with tie host or 
as a stand atone processor; some variation in the upper level s/w models exists. However, these variations 
serve to frriegrate fre various type applications go as to permit a single set of lower is vol functions to satisfy 

so all three applications. The explanation wi8 address the attached version SfW first and then the modifications 
required for standalone modes. 

tn any system, as frustrated by FIGURE 10, where the APAP is intended to arisen to a host processor 
the user's primary program would exist vrithin the host and would delegate to ihe APAP un* tasks end 
annotated data as needed to provide desired load balancing. The enctee of interpreting the dispatched 

ss task's program within ttw host or the Artery Director is a user option. Host l©v«4 interpretation permits the 
Array Director to work at interleaving users which do not exploit dose control of the Array! while APAP 
Fn^rpretation leads to minima) latency In control branching tut tends to limit the APAP tiw* to perform 
multi-user management tasks. This leads to the concept ehal fihra APAP and host can be tightfy or loosely 
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Two examples illustrate me extremes: 

i. When APAP i$ atieched to 309Q dees machines with Floating Point Victor Faciii&sa, user data in 
compressed term could be stored Yflthin the APAP. A host program that called for a vector operafion 

3 upon two vectors wtth differing sparseness characteristics would then send instructions lo Ihe APAP to 
realign the data into element by element matching pairs, output the result to me Vector Facility, read 
answer from the Vector PadJHy and Anally reconfigure data Intel final sparse data form. Segments of the 
APAP would be Interpreting and bulldn^ sparse matri* bit maps, while ofier sections would be 
eslcuferting how to move data between PMEa such that it would be properly afigned for the zipper 

70 2. With APAP attached to a smal inflight military computer, the APAP could he performing the entire 
workload associated with Sensor Fusion Processing. The host might initiate the process once, send 
sensor date as 3 was received lo ihe APAP and then wait kx results. The Array Director would then have 
to schedule and sequence the PME array through perhaps dozens of processing steps required to 
perfoim the process. 

rs The APAP will support thret levels of user contra* 

1. Casual User. Sflie worts with supplied routines, and hbrary [unction. These routines are maintained at 
the host or API level and can be evoked by foe usei via subroutine calls within his program. 

2. Custornlzer User, S.'he can write special tuncaons which operate within the API and which directly 
a vote routines supplied with the API or services supplied with the GO or PME 

2© 3. Development User. generates ptograms for execution in the CO or pivh, depending upon API 
servfcos for program load and status feedback. 

Satisfying ftese ttiree user levels in either closely of loosely coupled systems leads to Ihe partitioning 
of HW control tasks. 

2$ API Software Tasks 

The appfcation program Interlace API corrfatas S/W services thai can test the leading words of data 
received and can determine whether that defe should be Interpreted by the API, loaded to some storage 
within the Array Director or PM E, or passed to ihe I/O zipper. 

so For data that is id be Interpreted. Ihe API determines the required operation and invokes fte function. 
The most common type operation would cell ior ihe Array to perform some function which would be 
executed as a result of API wiles to ths CS (and indirectly to Ihe CC). The actual dais written to the CS GC 
would In general be constructed by the API operational routine based upon the parameters passed lo ms 
API from the host. Data cent to the CS/CC would in turn be forwaided to Ihe PME* w 9 the node BCI< 

as Date could be loaded to enher API storage, CC storage, or PftC memory. Further, due to be foacted to 
PME memory could be loaded via either the 1/0 zipper or via the node BCL For date to be put into the API 
memory, the booming bus would be read then written io storage. Data targeted to the CO memory would 
be similarty read and then be written *> (he CC memory, Fmafcy, data for the PME memory (in mis case 
normaBy new or additional MlMD programs* could be sent to all or selected PME& via the C$>CtfNode BCI 

<e or So & subset of PME* ior selective lediahfbuf on via the ro zipper. 

When dsrta is lo be sent to the MO zipper, re could be preceded by inline commands that permit the 
PME MlMD programs to determine its ultimate target or, it could be preceded by calls to the API service 
functions to perform either MlMD Initiation or SIMD transmission. 

In addition to responding to requests tor service received via fte host interface, the API program w3l 
respond to reouest from the PMEe\ Such requests will be generated on fte high speed serial port and w3l 
be routed through the CO'CS combination. Requests oSf this sort can re&uft in the API program's drrecfly 
servicing the PME* or accessing lha PME& via the standard speed serial port to determine further qualifying 
data relets *> the servtee request. 

so PME Software 

The software plan Includes: 

o Generation <rf PME resident service routines (that is, 'an extended ISA') lor complex operations end 
i/O management 

5s o Definition and development oi controller executed cubrougnes (hat produce and peso oontrd and 
parameter data to the PMEs via the BQt bus. These subroutines 
i, cause a set of PMEs to do mathematical operations on distributed objects, 
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2, provide IfO data management and synchronization sendees for PME Array and System Bus 
interacts*. 

3. provide startup program load, program overlay and program task management for PMCs. 
o Development cf <tete allocation support services tor Dost level programs, and 

a o Dmiopme^ ol a programming support system inducftDg assembler, simulator, and HW monitor and 
debug workstation. 

Baaed upon studies of mlfltary sensor fusion, optimization. Image ftansfarmatlon, US Post Office optical 
character recognition and FBI fingerprint matching applications, w® have concluded that a parallel processor 
programmed with vector and array commands BLAS calls) would fee effective. Hie underlying 
re programming modal must match the PME array characteristics feas&Js with today** tacKnoJogy. Spectfi- 
cetly: 

o PMEs can be Independent stored program processnra, 
o The array can nave thousands of PMEs, and be suitable tor tins grained p&sJteltem. 
o Inter-PME networks will nave very high aggregate bandwidth and a small logical diameter 4 , 
12 o Bui, by network connected microprocessor MIMD standards, each PME Is memory limited. 

Prior programming on MIMD parallel processors has used task dspatcfang metfiodoioay, Such 
approaches lead to each PME needing access to an portion of a targe program. This characiarlsile. In 
combination wai the non-shared memory characteristic of the hvw, would exhaust PME memory on any 
sicnBteanf problem. We therefor* target what we beltevs Is a new programming modal, called 
20 'asynchronous SIMD* (A-SIMD) type processing. In Ms conneeUon see U3SN 788,788, fried November 27, 
1991 of P. Kogge, which is Incorporated herem, 

A-SWD programming in our* APAP design means that a group of PMEs will be directed by commands 
broadcast to them as in 8IMD models. Hie broadcast command war initiate execution of a MIMD function 
wRhin each PME. That execution can involve data dependent branching and addressing within PMEs, and 
Z5 I/O based synchronization wrth either other PMEs or th» BCI. 

Normally. PMEs wffl complete the processing and synchronize by reading tfte next command from the 
BCI. 

The ArSiMO approach includes both MIMD and 8IMD operating modes. Since the approach Imposes no 
actual time limits on the command execution period, a pme operation that synchronizes on data transfers 

a> and executes indefinitely can be Initiated. Such functions are very effective In data filtering, DSP, and 
sy&teJio operations. (TThoy can be ended by either BO interrupts or by commands over the serial control 
bu30*4 $AMD operation results from any A-SIMD control stream *at does not include MIMD Mode 
Commands, Such a control stream can include any of the PMEs native Inatmciona These instructions are 
routed directry (o the instruction decode logic of the PME, Eliminating fte PME instruction fetch provides a 

35 htgher performance mode for tasks that do not invoke data dependent branching. 

This p^ogmwning model (supported by H.W featuresj extends to psrmming the array of PMEs to be 
divided into Independent sections. A separate A-SIMD command stream controls each section. Our 
application studies show that programs of interest divide into separate phases input input buffering, 
severe! processing steps, and output formating, etc), suitable for pipeline data processing. Fine-grained 

40 paralfeUBm results from applying the n PMEs In a section to a program phase. Applying coarae-gralnod 
partitioning to applications often results in discovering small repetitive tasks suitable MIMD or memory 
bandwidih limited tasks suitable lor S1MD procoesHg. We program the MIMD portions using conventional 
techniques and program the remaining phases as A-SlMD sections, coded with vectorized commands, 
sequenced by the airay contiolter, This makes the large controller memory the program store. V&ying the 

45 number of PMEs per section pesmfes balancing the workload. Varying the dlGp&ched iask b*zo permits 
balancing the BCI bus bancfartdth to the control requirements. 

The programming modal tfso oonefdara allocating data elements to PMEs. The approach is to distribute 
dala etemente evenly over PMGs, In eorly versions of this will be done by fihe programmer or by 3AV. 
We recognize that the IBM parallelizing compiler tccrmologtes apply to this problem and we expect to 

so Investigate their usage. However, the irter^PME bandwidth provided Goes tend to reduce the importantly or 
this approach. Tnis links dala allocation and I/O mecharssm performance-. 

The H/W requires that the PME Initiate data transfers out of Its memory, and it supports a controlled 
write into PME memory wrShoul PME program involvement Input control occurs in the receiving PME by 
providing an input buffer address and a maximum length. When I/O to a PME results In buffer overflow, 

sg HrW will interrupt the receiving PME. The to** level I/O functbfts ftat wiB be developed for PMEs build on 
this service. We will support stirrer movement of raw data botwoen adjacent PMEs or movement oi 
addressed data between any PMEs. The last capability depends upon the circuit switched and store and 
forward mechanisms. The interpret address and forward operation is important for performance. We have 
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optimized the H/W and &W to support the operation. Using one word buffers results in an interrupt upon 
receipt of address header. CoYnpsring target id *lth local Id permits output path selection. Transfer of the 
subsequent date wonte occurs in circuit switched mocte- A slight variation on thte process using Kur^r 
buflere results In a store and forward mechanism. 

9 Because of the high performance Inter-PME bandwidth. It is noi always necessary or desirable to place 
data etomertis witttln the PME Array carshjly. Consider shifang a vector daia element distributed across 
PMEs. Our architecture can send daxa without an address header, thus, providing for very fast I/O. Htwttver, 
we have found, in many applications, that optimizing a data structure for movement in one direction, 
penalizes data movement in en oittogorwl direction. The penalty in such amiaiions approximates tns 

10 average cost of randomly routing data in the nefaork. This leads to applications where placing data 
sequentially or randomly {es opposed to arranging dalai resufts In shorter average process t*me&. 

Many applications can be synchronized to take advantage of average access time. (For example, POE 
relaxation processes acquire data from a neighborhood and thus, can average access over at least four LO 
operations-} We betteve that after considering the factors applicable to vector and array processes, 1*9 
15 scaiterrgether or row'column arrihmetfc, many users wilt 5nd brute force daia allocation to be suitable for the 
application. However, we know of examples that Illustrate application characteristics (Tike required synchro- 
nization or biased utilization of shift directions^ that tend to force particular data allocation patterns. Thb 
characteristic requires that the tools and techniques developed support eiSwr manual tuning of the data 
placement* or simple and non-optimum data allocation- (We wi9 support the non -optimum data alocation 
so strategy wifli host level macros to provide near transparent port of vectorized host programs to the MRP. 
The KW Monitor workstation will permit the user to investigate the resuham performance.} 

FIGURE 10 shows the general S/W development and usage environment The Host Application 
Processor is optional in that program execution can be controlled from either the Host or me Monrtor. 
Further, the Monitor will effectively replace the Array Controller is some situations. The environment will 
support program execution on real or simulated MRP hardware. The Monitor Is scenario driven eo that the 
developer doing test and debug operations can create procedures to pemvt effective oper*6on at any level 
of abstraction. 

RQURE 20 illustrates me levels of H/W supported within the MPP and the user interfaces so these 
levels. 

so We see two potential application programming techniques far the MPP. In the least programmer 
intensive approacK eppftceaona would be written in a vectorized high order language, if the user did not 
feel that the problem warranted tuning data placement frsn he would use compito time service* to allocate 
data to the PME Array. The application would use vector cafe like BLAS Ital would be passed to the 
controller for interpretation and execution on ths PME Array. Unique cafis would be used to move data 

3S between host and PME Array. In summary, the user would noi need to be aware of how the MPP organised 
or processed the data, two optimization techniques will be supported for this type application: 

1. Altering the date allocation by constructing the data allocation table will permit programs to force data 
placements. 

2. Generation of additional vector commands far execution by the array controfier wBl permit tuned 
<o suMmctions fie. calling the Gaussian ERrnJnaiion as a single operation.) 

Ws also see thai the processor can be applied to specialized applications as in those referenced in tha 
beginning of this section, m such cases, code tuned to the application would be used. However, even in 
such appttcattens the degree of tuning wHI depend upon how important a particular task is to the application. 
It Is In thrs situation that we see the need for tasks individually suited to SI MO. MiVlD or A*8lt¥ID modes. 

4* These programs win use a comwnason oft 

1. Sequences ol PME native instructions passed to an emulator function Within the array controller. The 
emulator wlB broadcast the Instruction and tts r parameters to the PME est The PMEs in this 6IMD mode 
win pass the instruction to fro decode function, simulating a memory fetch operation. 

2. Tight Inner loops that can be ifO synchronised will use PME native t$A programs. Alter initiation from 
aa a SIMD mode change, they would run continuously m WMD mode. (The option to return to SIMD mode 

via a 'RETURN* inssruction exists.) 

3. More complicated programs, as would be written In a vBcrorfrtng command set, would execute 
subroutines in the array controller that Invoked PME native functions- For example a simplified array 
comro8er program to do a BLAS *SAXPY r command on vectors loaded sequentially across Pfctes would 
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' Gaussian Eliminacian wlih normal pivoting raoulres shifting rows bui not columns. More than ?n performance 
difference would result from arranging rhe data such that columns were on the fast shin direction. Evan wlrh 
Lhal there is noi an advantage to arranging rows in any particular relationship to the buses. 
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start sequencss wahln the PMEs that: 

a. Enable PMEs with required * elements via comparison of PME Id *tlh broadcast Tnox' and 
1 x_eddr- values, 

b. Compress the? x values via a writs to consecutive PMEs, 

5 c Calculate I he address oC PMEs wtlh y Yemenis from broadcast data, 

c. Transmit tna compressed x data to the y PMEs, 

9. Do a single precision floating point operation In PMEs receMng x values to complete the operatfon. 
Rnalry* the SAXPY example illustrates one adcfitlonal aspect of executing vectorised eppScetk>n 
programs The major steps execute m the API and could be pmgremmed by eHher an optimizer or product 

?o developer. Normal/, the vsctorired appiication would call rather than tncbde this level o coda. These steps 
would be written as C or Fortran ooda and wBl use memory mapped read or writes la control the pme array 
via the BCI bus. Such a program operates the PME array as a eenea of M1MD steps synchronized by 
returns to !he API program. Minor steps such as the single precision floating point routines would be 
developed by tte Customlaer or Product Developer. These operations will be coded using the native PME 

« ISA and will be tuned to the machine characteristics, si general, ftls v/ould be the domain of the Product 
Developer since coding, last and optimization at this level require usage ol the complete product 
development tool set 

The APAP can have appfcauons written in sequential Fortran. The pain rs quite dnfererrt. FIGURE 21 
outlines a Fortran compiler which can be used. In the first step. It uses a portion of the existing parallelling 
70 compile* to develop program dependencies. The source plus these tables become an input to a process 
that uses a characterization of the APAP MMP and the source to enhance parallelism. 

This MMP is a non-shesred memory machine and as such allocates data between the PMEs tor local 
and global memory. The very fast data transfer time* and the high network bandwidth reduce the time 
affect ot date allocation, but ft stiB is addressed. Our approach treats part of memory as global and usee a 
a$ $w service funcSoo. It is else possible to use ths dependency information to perform the data allocation in 
a second aKemsrave. The final step k\ converting the source to muffipte sequential programs is performed 
by the Level Partitioning step This partitioning step is analogous to the -f Fortran sup 3:ef work being 
conducted wtth darpa funding. The last process In the compilation is generation ol the executable code a? 
all individual functions! levefe. For the PME this will be done by programming the cede generator on an 
a» existing compiler system. The Host and API code compilers generate the code targeted to those machines. 
The PME can execute MIMD software from ito own memory. In general, the multiple PMEs would not 
be executing totally different programs but rather mould be executing the same small program in an 
asynchronous manner. Three basic types el S/W can be considered although the design approach does not 
Rrrtft the apap to just these approaches: 
as I. Specialized emulsion functions would make fte PME Array emulate the set of services provide by 
standard user libraries nice UNPACK or VP$$, h such an emulation package* the PNffi Array couW be 
usmg its multiple set of devioes to perform one of m operations required m a normal vector caa. This 
type of emulation, when attached to a vector processing unit could utilze the factor unit (or some 
operator* while performing others internally. 
4> a The parallelism os the PME Army could be exploited by operating a set of sofbvare that provides a 
new sat of mathematical and service functions in the PMEs. This set of primitives would be the codes 
exploited by a customizing user to construct his eppifc&tion. The prior example of performing sensor 
fusion on a APAP attached to a military platform would use such an approach. The customlzer would 
write routines to parlor m Kalman FBtars, Track Optimum Assignment and Threat Assessment using the 
<* supplied set of function names. This application *otild be a series of API cai statements, and each call 
would result in initiating 8ie PME set to perform some basic operaaon like 'matrix multiply* on data 
stored within the PME Array. 

3. In cases where no effective method, considering performance objectives, oi applicafion needs exists 
then custom S/W could be developed end executed within the PMC. A specific exarvple is 'Sorf. Many 

so methods to sort data exist and the objective in all cases Is to tune the process and the program to the 
machine architecture. The modified hypercuDd is well suited to a Batcher Sort; however, that sort 
requires extensive calculations to determine particular elements to compare versus very short compari- 
son cycles. The computer program in FIGURE 17 shows a simple example of a PfcE program 1 100 to 
perform the Batcher Sort 1000 wsh one element per PME. Each one or the program description would be 

bo expanded lo 3 to 6 PME machine level instructions, end s8 PMEs would even execute the program in 
M1MD mode- Program synchronization is managed vie the I/O statements. The program extends to more 
dale elements per PME and to very large parallel sorts in a quite straight toward manner. 
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CC Storage Contents 

Date from the CC sfoiage is used by the PME Array in one of two manners. Whan the- PMEs w 
operating in 3MD, a series of instructions can bo fetched by the CC and passed to the nod* BC1. thus, 
5 reducing load on both the API and C$. Alternatively, [unctions thai are not irequerrtly required, such as PME 
Fault Reoonflgu?«?ton SW. PME Diagnostics, aid perhaps conversion routines can be stored in trie CG 
memory. Such functions can then be requested by operating PME WIMD programs or moved to toe PMEs 
at the request of API program directives* 

10 Packaging of the frWay Modified Hypercubs 

Our packaging techniques tate advantage of the eight PM6s packaged in a sfngte chip and arranged in 
a N-dfirnertsionaJ modJied hypercuba configuration. This chip level package or node of the array is tfia 
smallest budding Hoe* In the APAP design. These nodes ere *ieo packaged In en 8 X a array where the 

rj X end the +-Y metes ring3 wHNn ihe enrey or cluster and the *-W, end +*Z em brought cut to fris 
neighboring clusters, A grouping of clusters make up an array. This step significantly cuts down wire count 
for data and control tor the array. The W end Z buses will connect to the adjacent dusters and font) w &kI 
2 ring? to provide total connectivity around the completed array of various efts. The massively parallel 
system vH2 be comprised of fliaso cluster building blocks k> form the massive array of PMEs. The APAP 

w vA\\ consist of en 6 X 6 array of casters, each cluster will have Its own controller and ell the controfie** w'l 
be synchronized by our Array Director. 

Many trade-efts of wireebtiity and topology have been considered, yet wtth these considerations we 
prefer the configuration which we illustrate *tth this connection The concept disclosed has the advantage of 
keeping the X end V dimensions within a cluster level of packaging, and distributing the W and Z bus 

so corm ectlons » all the neighboring clusters. 

After rmtfsmentfog the techniques daacrfcfcd, the product nill be wireabte. and manufacture white 
maintaining the inherent characteristics oi the topology defined. 

The concept used here Is to mix, match, and modify topologies at different packaging levels to obtain 
the desired results in farms of wire count For the meftod to define the actual degree of modification of me 

30 hyperoube, refer to the Rolfe modified hypercubo patent application referenced above. For the purpose oi 
this preferred embodiment we will describe two packaging levels, to simplify our ascription, it can be 
expanded. 

The fi?Bt Is the chip design or chip package illustrated by FIGURE 3 and FIGURE 11 . There are eight of 
the processing elements wHh their associated memory and oommunic^ion logic encompassed into a single 

38 chip which Is defined as a node. The internal configuration Is classified as a binary hypercube or a 2-degree 
hypercubo where every PME is connected to two neighbors. See the PME-PME communication diagram in 
RGURE 9. especially 500, 510, 52Q, 530. 540. 550, 560. 570. 

The second step is that the nodes ere configured as an ft X 8 array to make up a o loafer. The fuQy 
populated machine is bu»t up of an array of 8 X 0 clusters to provide the maximum capacity of 32738 

<o PME*. These 409$ nodes ere connected m an 8 degree motffted hypercube network where the commu- 
nication between nodes is programmable. This ability to program different routing paths adds flexfoifcy to 
transmit different length messages. In addition to message length differences, there are algorithm optimiz- 
ations that can be addressed with these programmaJrility features. 

The packaging concept is intended to signhlcantly reduce the off page wire count tor each of the 

4* clusters. Tins concept takes a cluster whtcn le defined as a a x e array of nodes aaa each node 825 having 
S processing elements for a total of 512 PMEs, then to limit the X and V ring within the cluster &kJ, finally, 
to bring out the W and Z buses to aD clusters. Tha physical picture could be envisioned ae a sphere 
configuration €00, $i0 ot 04 smaller sphe»es 330- See FIGURE 10 for e future peckaging picture which 
illustrates the iuJJ up packaging technique, limiting the X and Y rings 800 within the duster end extending 

so out the W and Z buses to all clusters S10. The physical picture could be envisioned as a sphere 
configuration of 64 smaller spheres 830, 

The ectusJ connection of e single node to the adjacem X and Y neighbors 375 exists within the same 
cluster. The wiring savings occurs when the z and w buses are extended to the adjacent neighboring 
clusters as fflusetfed tn FIGURE 16. Also illustrated In RGUFE 10 is the m of the chips or nodes that can 

56 be configured as a sparsely connected ^dimensional hypercube or torus 600, £05, 910, 9i& Consider each 
of the 8 external porta to bo labeled as +X, + Y« +z« +w, -x, -Y, -z, -w 950. &75. Then using m chips, e 
ring can be constructed by connecting the +x to -X pons. Again m such rings can be Interconnected into a 
ring of rings by interconnecting the matching + Y to -Y ports. This level of structure will be called a cluster. 
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ft provides tor 512 PMEs and will be the building block for several size systems. Twq such connections 
(950, 075) are shown In the diagram flluatrated In FIGURE 10. 

AppScafons tor Ogsksjjfe MPP. 

5 

Hid deskside MPP si a workstation can be effectively applied In severe! application areas InclucEnrj 

1, Srnafl production tasks that depend upon compute intensive processes. The US Postal Service 
requires a processor that can accept a fax Image of a machine printed envelope and then find and read 
the zip code, The process Is needed at all regional sort iscltlttee and is en example of e very repetitive 

70 but suU compute intensive process. We have Implemented APL language versions of a sarnpte of the 
required programs- These models emulate the vector and &f&; processes that will be used to do the 
work on the MPP, Bessd upon twa test, we Know that the task Is an excellent match to t* processing 
ajcniteotune. 

2. Tasks In which an analyst ea a reeuft of prior output- or expected needs requests sequences of data 
95 transformations. In an example taken from the Defense Mapping Agency, satellite images are to be 

transformed and smoothed pbtel by pixel Into some other coordinate system. In such a sttaatfon. the 
transformation parameters tor the image wary across localities as a result of ground elevation and slops. 
The analyst must therefore add fixed control points and reprocess transformations. A rimflar need occurs 
in the utilization of scientific simulation results '/men users require almost real time rotation or pempec- 
20 tjv* changes. 

& Program development for production versions of *e MPP will use workstation bob IYIPPs, Consider a 
tuning process that requires analysis or processor tferaus nobvork performance. Such a tasft is rnacftine 
and enaJytf interactive, K can require hours when the machine is idle and analysi is working. When 
performed on a supercomputer U would be very costly. However, providing an affordable workstation 
at MPP wttb the same (but seated) characteristics as the supercomputer MPP eliminates eoets and eases 
the feat and debug process by eliminatino me programmer inefficiencies related to accessing remote 
processors. 

FISURE 22 is a drawing of the workstation accelerator. It uses the same stee enclosure as the 
RISC/GCOO model 590. Two ewing out gates, each containing a fun cluster ere shown. Tne combined two 
ao clusters provide 5 6 OPS of fixed point performance and 630 MftopS of processing power and aboul 100 
Mbyiefe of I/O bamfcriOi to the array. The unit would be suitable for any of the prior applications. With 
quantity production and inefudina a host RISCeQOQ, ft would be price mrnparabie with high performance 
workstations, not at the price of comparable machines employing old technology. 

as Description of the AWAC3 Sensor Fusion 

The mHfrary environment provides a series of examples shown© the need for a hardened compute 
Intensive processor. 

Communication in the targeted noisy environments implies the need tor digitally encoded communica- 

40 lions, as is used in ICNIA systems. The process of encoding the date for transmission and recovering 
information after receipt is a compute intensive process. The task can be done with specietized signal 
processing modules, but for situations where comrminrcatJon encoding represents bursts of aetrvtty. 
special zed modules are mostly idle. Using the MPP permits several such tasks to be allocated to a single 
module and saves weight power volume and cosl 

is Sensor data fusion presents a particularly dear example of enhancing an existing p&atform with trie 
compute power gained from the addition of MPP, On tie Air Force E3 AVVAC9 there are more than tour 
sensors on the platform, but there la currently no way lo generate tracks resulting from the Integration of all 
available data. Further, ft& exisSng generated tracks have quite poor quality uue to sampling characteristics. 
Therefore, there is motivation to use fusion to provide an effect v© higher sample rale. 

so We have studied this sensor fusion problem in detail end can propose a verifiable and effective solution, 
but that solution would overwfiesn the compuie power available in an AWACS data processor. FIGURE 23 
shows the traditional track fusion process. The process is faulty because each of the individual processes 
lends to make some errors and the final merge lends lo collect them Instead o* eliminating them. The 
process fe else characterized by higft time latency in that merging does not complete untf the slowest 

sq sensor completes. FIGURE 24 presents an improvement and the resulting compute intensive problem with 
the approach. Although ue cannot solve a NP-Hard problem. have developed a good method to 
approximate tte solution. While m details of that application ere being described by the inventors 
elsewhere, as it can be employed on a variety cf machines lifee an Intel Touchstone with 512 i860 {60360) 
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processors, and IBM's Scientific Visualization System, il can be used as an applicoeon suitable for ths MMP 
using the APAP design cteBcrtDed here wan cay 128.000 PMEs. suastanllalry outperforming these other 
systems. Application experiment* shew the approximation quality i* below the level of sensor noise and ae 
such the answer Is sppBcabfo to applications like AWACS- RGUR6 25 shows the proceed ng loop involved 

5 In (lie proposed Lagrengean Reduction n-dlmension^ Assignment algorithm. The problem uses very 
coftirtited repetiQons of {he wait known 2-dimen*k>nai assignment problem the earns algorithm tliat 
classical sensor fusion processing uses. 

Suppose for example ftat the n-dimenstonal aloortthrn >*es to to applied to the seven sets of 
observations illustrated in FIGURE 24 end further, suppose that each pass through a reduction process 

10 required four iterations through a 2d Assignment process. Then the new fid Assignment Problem would 
require 4000 iterators of t>e 2d Assignment Problem. The AvYAC$' workload is now about 90% of machine 
capac&y. Fusion perhaps requires 10% of the total effort but even that small effort when seated up 4000 
times results in total utifizalion bs^ng 370 times toe capacity of an AWACS. Not on?/ does this workload 
overvfheJm the existing processor, but it would be marginal In any new ML environment suited* coarse* 

;s grained, parallel processing system currently existing or anticipated in the next few years. K the algorithm 
required an average of 5 rather than 4 iterations per step, then A would overwhelm even the ItypothesUed 
systems. Conversely » the MPP solution can provide the compute power and can do so even at the 6 
Ko ration level. 

6 Mechanical Packaging 

As Illustrated in FIGURE 3. and oilier FIGURES, our preferred chip is configured In a quadflaipaek form. 
As such it can be bricfcwalled into into various 2 O and 3 O conffcuranons in a package. One chip of eight or 
more processor memory eJsmenis is a first level package module, the same as a single DRAM memory 

20 chip Is to a foundry which packages the chip. However, It Is In a quecflatpactt form, allowing connections to 
one another in four dictions. Each connection is point to point. (One chip in its first level package is a 
module to the foundry.) We are able 1o construct P£ arrays of sufficient magnitude to Nl our performance 
goals due to this feature- The reanry Is shai you can connect these chips serosa 3> 4 or even five feet point- 
to-poim, ie. mufthprccessor node to node, and &w have proper control without the need or fiber opffcs, 

3D This has an advantage for the drive/receive circuits that are required on fte modules. One can achieve 
high performance and keep the power dissipation down because we do not have bus Systems that daisy 
chain from module to module. We broadcast from node to node, but Ms need not be a high performance 
path. Most data operations can be conducted in a node, so data path requirements are reduced, Our 
broadcast path is essentially primarily used as a controller routine tool The data stream attaches to and 

3S runs in, the ZWXY communication path system. 

Our pew dissipation is 22 watts per node module for our cooimerclal workstation. Tli<9 allows us to 
use air cooled packaging. The power system requirements tot our system aie also reasonable because of 
this fact. Our power system illustrated muJupfes the number of modules supported by about 2.5 watts per 
module, and such a five volt power supply Is very cost effective. Those concerned with the amount of 

40 electricity consumed would be astonished that 32 microoomputars could operate wfh lass than iha wattage 
consumed by a re ad tog 6ght 

Our thermal design b enhanced because of the packaging. We avoid hot spots due to high dissipating 
parts mixed wlm tow dissipating ones. This reflects dtrecfiy on the cost of the assemblies. 

The cost of our system Is very attractive compared In the approaches that put a superscalar processor 

« on a card. Our performance level per assembly per watt per connector per pan type per dollar 13 excellent 
Furthermore, we do not rwsd tie same number of packaging levels that the other technology does. We 
do not need moduk>A^rd^bsokptane and cable. We oan skip the card level if we want tow Ae Illustrated In our 
workstation modules, we have skipped the card level wriih our brickwaJJed approach. 

Furthermore, as we illustrated in our layout, each node housing which is brrckwalled In the workstation 

to modules, can as illustrated in FIGURE 3 comprise multiple replicated dtes. even wttltin ins same chip 
housing. While normally we would place one die within an air cooled package, it is possible to place 8 die 
on a substrate using a multiple chip module approach. Thus, the envisioned wazch with 32 or more 
processors. Is possible, as well as many ofrer sppflcations. The packaging and power and flexibility make 
applications which are endless, A house could have its controllable tistrumenta eil watched, end coordl- 

59 naiad with a very small pari Those many chips lhal are spread around an automobile for engine watching, 
broke adjustment, and so on could all have a monitor wtthio a housing, in addition! one the same substrate 
with hybrid technology, one could mount a 388 microprocessor chip with full programmable capabO&y and 
memory mil in one chip) and us* it as the array controller for ihe substrate package. 
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We have shown many ccnf^rafians or systems, from control systems, FIGURE $> to larger and larger 
gyetems. The atflfty to package a chip with multiple processor memory element ol eight or more on a chip 
in a (lip, with pinouts fritjng In a standard DRAM memory modute, such *$ in a SIM module make possible 
countless additional applications ranging from centrals to wall size video displays which can have a 

s repetition rate, not a the 15 or ao frames that press the exlsflng technology today, but at 00 frames* wltti a 
processor assigned to monitor a pixel, ot a node omy a lev* pfcete. Our brick-nail quediiatpacfc makes it easy 
to replicate ths same part time over and over again. Furthermore, me replicated processor Is reaRy memory 
with processor interchange. Part of the memory can be assigned to a specific monitoring task, and anotter 
part (with a size pros^nrirfcalry defined) can be a massive global memory, addressed po*nt-*o-polnt, with 

jo broadcast to all capability. 

Our besk? workstation, our supercomputer, our contrgler, our AWA0$. en are examples of packages 
tha can em^oy our new technology. An array ol memory, wlm inbuilt CPU chrpe and 1.0. functions as a 
PME of massiwly parsSel applications and even more Smiled applications. The llextbiRty ol packaging and 
programming makes bnoglngrtws expand and our technology allows one part to be assigned to many Ideas 

is and images. 

MMftary Awtontes Appicatlons 

The cost advantage of constructing a MIL MPP Is particularly wen illustrated by the AWACS. Il is a 20 
no yw old enclosure t^af Has grown empty space as new technology memory modules have replaced the 
original core memories- FIGURE 26 shows a MIL qudrfrable iwo clusrer system that would fit directly into 
the rack's empty space and would use the existing memory bus system for [reconnect ton. 

Although ths AWACS example is very advantageous due to the existence ot empty space, in other 
systems H Is possible to create space. Replacing existing memory w»h a smell MPP or gateway to an an 
29 Isolated MPP Is normally quite viable. In such cases, a quarter cluster and a adapter module would result In 
a 8 Megabyte memory plus 640 MIPS and use perhaps two slots. 

Supercomputer Appftcetton 

a> A 64 cluster MPP Is a 13.6 Gflop cupercom puter. H can be conRgured in a system described in FIGURE 
27. 

The system we describe allows node chips to be brick waited on cluster cards as illustrated in PK5URE 27 
to btiid up systems wltti some significant cost and size ach/antages. There Is no need id include extra chips 
such as a network switch in such a system because it would increase costs. 

oe Our interconnection system with "brickwalled" chips allows systems to be built like massive DRAM 
memory is packaged and wEi have a deined bus adapter conforming to lie rtgW bus specifications, for 
instance a microtiiannsJ bus adaptor. Each system wifl have a smaller power supply system and cooling 
design than other systems based upon many modern microprocessors. 

Unlike most supercomputers our current preferred APAP with Hosting point emutafeon Is much faster In 

40 integer arithmetic (164 Ol PS} s is when doing fleeting point arithmetic. As such, the processor would 
be most effective when used m applications thai are very character or integer intensive. We have 
oonsldered three program challenges which m addition to the other applications discussed herein are 
nsediui ot solution. The applications which may be more important *an some of tfta "grand challenges" to 
day to day life includs: 

<* 1 . 3090 Vector Processors contain a very high performance floefrg point arfthmcnc unit. That unit as do 
most vectorized floaang point units, requires pipeline operations on dense vectors. Applications thai 
make extensive use of non-regular sparse matrices fl.e* matrices described by bit maps rather than 
diagoneis) wste the performance capability of the floating point unit The fcff>P solves this problem by 
providing the storage for the date and using its compute power and network bendwidlh, not to do the 

so calculation but rather to construct dense vectors, and to decompress dense results. The Vector 
Processing Unit is kept busy by « cortiinua! flow of operations on dense vectors being supplied to it by 
the MPP, By sizing me MPP so tfar. it can effectively compress and decompress at the same rate the 
Vector FaciEly processes, one could keep both units tony busy. 

2, Another host stashed system we considered is a solution to the F6I fingerprint matching protHem. 
ss Mere, a machine with more then 04 clusters was considered. The problem was to match about 6000 
fingerprints oer hour against the enSre database of fingerprint history, using massive DASD and the full 
oandwidih of the MPP to host aaachment, one can roil the complete data base across the incoming 
prints in about 20 minutes. Operatfng about 75% ol the MPP in a 8IM0 mode coarse matching 
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operation, balances processing to required throughput rate, We estimate that 15% of the machine to A- 
SIMD processing mods would (hen a*np!ete me matching by doing (ha detailed verification of unknown 
print versus file print for esses pacing toe ooar** filer operation. The remaining poisons of the machine 
were in MIMD mode end altooated to reserve capacity, work queue management and output formatting. 

9 3. Application of the MPP to database operations has been considered. Although the work b very 
preliminary, It doss seem to be a good match. Two aspects ot the MPP support this premise: 

a. Tho connection between a duster Controller and the Appllcetion Processor Interfaca is a Micro- 
Channel. As such, It could be populated vdth DASD dedicated to the cluster and accessed directly 
from the cluster, A tM duster system with six 540 Mbyte hard drives attached per cluster would 

10 provide 246 Gbyta storage. Furlhar, that entire database could be searched sequentially in 10 to 20 
seconds. 

b. Databases are generally not searched sequentiairy. Instead they use many levels of pointers. 
Indexing of databases can be done w&hin the cluster. Each bank of DASD would be supported by 2.5 
SIPS of processing power and 32 Mbyte of sfcrage. That sufficient for bo*i searching and storing 

;a ir» Indices. Snce indices are now frequently stored within the DASD, aignifrcare performance gains 

would occur. Using sudi an approach and dispersing DASD on SCSI Interfaces attached to the cluster 
MicroChannel permits effectively unlimited size data bases, 
FIGURE 27 Is an illustration ot the APAP when used to buttd the system into a supercomputer scaled 
MPP. The approach revert* to replicating units, but here it is enclosures containing 1ft dusters that are 
so replicated. The particular advantage of $*$ replication approach is that the system can be scaled to «urt the 
user's needs. 

System Architecture 

£s An advantage of the system e/chitecture which is employed in the current preferred embodiment la the 
ISA. system which will be understood by many who wHI form a pool tor programming the APAP, The PME 
ISA consists ot ihe folkm-tng Data and Instruction Formate, illustrated in the Tables. 

Data Formats 

so 

The basic (operand) size is the 16 bit word. In PME storage, opererfcds are located on integral word 
boundaries. In addition to tfw word operand stea, other operand sizes are avaltetfe in rnufiipies of 1 6 bits to 
support additional lunctiona 

Within any of the operand lengths, the bit positions of the operand are consecutively numbered from 
as left to right starting wi*> the number 0. Reference to high-order or most-significant bile always reier io the 
JefHrwt Wt portions. Reference to the few-order or leest-slgnlflcani bits always refer to (he right-most Wt 
positions. 

Instruction Formats 

40 

The length of an instruction format may either be 1$ bits or 32 bits. In PME storage, instructions must 
be located on a 16 bit boundary. 

The following general Instruction formats are used. Normally, the fret four bits of an Insirucilon define 
the operation code and are referred to as the OP bits, ki some cases, additional bits are required to extend 
4» the dettn&ion of tie operation or to define unique concmone which apply to the instruction. These bits are 
referred to as OPX bte. 


so 
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jo 


Format Code 

Operation 

RR 

R*giste* to Regisier 

DA 

Direct Addre&s 

RS 

Register Storsga 

Rl 

Register Immediate 

SS 

Storage to Storage 

SPC 

Special 


All formats have one field in common* This f teld and its tnlsipretation is: 

B3& 0-3 Operation Code - TMs fleld> oonv^mes in conjunction with en operation code extension 

field, defines the operation to be performed- 
Detailed figures of lha individual formats along wtln interpretations of their fields are provided In the 
toftowfog subsections. For some iftsattftfons, two formats may bo combined to form variations on the 
Instruction, These primarily Involve the addressing mode for the Instruction As an example a storage to 
storage Instruction may hew e term which Involves direct addressing or register addressing. 

FIR Fvm»at 

The Reglster-RegfSier <RR) format provides two general resteer addresses and Is id bas to length as 
shown. 


3t> 


OP 

( 1 1 

RA 

1 i 1 

Q Q 0 0 
1 1 1 

RB 

1 1 • 

Q 3 

4 7 

S 1 
1 

1 1 
Z 5 
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In addition lo an Operation Code field* the RR format contains: 

Bfts 4-7 Register Address I - The RA field is used to specify which of the 16 general registers is 
to be used as an operand anxtor destination. 

Zeros - BJt 6 being a zero defines the format to be a RR or DA 6orma5 and bits 9»n 
equal to zero define m oper aeon to be a re^teter to register operation <a special case of 
the Otreci Address formal). 
12-15 Register Address 2 - The RB field is used to specify v*lch of the 16 general registers Is 
to be used as an operand. 


<K DA Format 


The Direct Address <DA) format provides ana general register address end one direct storage address 
as shown. 


S3 
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OP 

1 1 1 

RA 

1 1 1 

0 

Dia AODR 
t i i i i i 

0 3 

<J 7 

8 9 1 


5 


In addition to m Operation Coda fob, sis DA format conning: 

BBS 4-7 Register Address 1 - The RA field is used Id specify which of tr^e 18 general rasters e 
to be used as so ope?end and/ior destfnetion. 
js B& 8 Zero - This bit being zero dslnes the operation to be a direct address op$ra»on or a 

register to register operation. 
Bits 9-15 Direct Storage Address - The Daect Storage Address field is used as en address into 
the level unique storage block or the common storage block. Bits 0-11 ot the direct 
address Held must be nomrero to deSne the direct address form. 

so 

RS Format 


Ths Register Storage <RS) formal provides or>e general register addressee end en indirect storage 
address* 

£0 


OP 

RA 

j 

OEl 

RB 

f 1 1 

i r i 


J L 

II i j 


9 3 4 7 8 9 11 1 
12 5 


fen addition to an Operation Code Held, trie RS format cc* *a?ne: 

Bits 4-7 Blister Address 1 - The RA field is used to specify vrtncb of *o 1 6 general registers 

to be used as en operand end/or destination. 
Bit 8 On* - This b3 being one defines the oo^ratFon to be * register storage operation. 

Bite £-11 Register Data - These bite are considered a Banned value which is used to modify the 

contents ot register specified by the RB field. 
Bits 12-15 Register Address 2 • The R8 fold Is used to specify which ot the 16 general registers te 

to be used as an storage address for an operand. 

Rl Format 


Tbe Register-lmnr.edi&te (R») lormet provides one ge^rd register address and ie bits of immediate 
dste, The Rl fcrmet la $2 bite of length ss shown: 

so 
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s 

OP 

ftA 

1 

DEL 

0 0 9 6 

IHMtOIATE MTA ... 


I 1 1 

- i i 1 


1 1 

1 1 4 

-UL.U..1 I'll 


3 4 


7 8 9 


to 


1 1 

1 Z 


1 

5 


a 
l 


m addition to an Operation Cods Held, the Hi format contains: 
s*7 


J3 


20 


9-11 


12-13 


Bits 16-31 


aj 9$ Format 


Register Address 1 - The RA fleid is used » specify vmlch of the 16 general registers is 
to be used as on operand end/or destination. 

One • Hire bti belrcg one defines Ine operation 1o bo a register storage operation. 
Register Data - These bits ore considered a signed value nh'tch e$ used to modify the 
cantoris of the program counter. Normally,tht3 field vrouid have a value of one for the 
raster Immediate formal. 

Zero** - The wetf be^o zero is used to specify the updated program counter, which 
points to the immsdiaiB date field, is to be used ee si storage address far en operand. 
Immediate Data - TNs tteW series as a 16 bit immediate data operand for Register 
IrnmedUse tastiucaons. 


30 


The Storage to Storage (SS) format provides two storage addresses, one expfcrt; and (he second 
Implicit The Implied storage address Is contained tn 6enera3 Register 1, Register 1 Is modified during 
execution of the instruction. There are two forms ot a SS instruction, a ffirect address fom> and a storage 
address Form. 



0F>X 











(Direct 

OP 


0 

c 


dir mm 

Address 


0 

V 

R 



Form) 


rl 

F 

y 




I f 1 





MINI 



0 3 4 7 8 9 1 

5 



OPX 













(Storage 

OP 


0 

c 

1 

DEL 

RS 

Address 


0 

V 

R 




Form) 


p 

r- 

r 

V 





-J. 1 t 





1 1 

\ \ \ 



0 3 4 7 8 9 11 1 
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In addition to on Operation Code field, the 33 formai contains: 

BUs 4-7 Operation Extension Code - The OPX Bold, together wkh mo Operation Code, defines 
?»e operation lo be performed. Bfc 4-5 define the operation type Such as ADD or 
SUBTRACT, BHs 6-7 control the carry, overflow, and how the condition codo wiH be set. 
Bft 6 = 0 Ignores overflow* bit 6 = 1 allows overflow. Bit 7 » 0 Ignore the cany stat 
during the operanon; bit 7 » 1 includes the carry sfat during the operation. 
Bit 6 Zero - Defines the form to be a direct address form. One - Dofinss the form to be a 

storage address form. 

Bits Direct Address (Direct Address Form} - The Cttrect Storage Address told is used as 

an address into Ihe level unique storage block or (he common storage* block. Bite 9-1 1 of 
the direct aoVfress field must be non^ro to deflne the direct address form. 

Bits 9-11 Register Delta (Storage Address Form) • Those bits fire considered a signed value 
which is used to modify the content* of register specified by the RQ Bold. 

Ba» 12*15 Register Address 2 Storage Address Fgnn) • The RB field le used to specify which 
of the 18 general registers is to bs used ea a storage address for en operand. 

SPC Formal 1 

The Special (SPC1) formal provides one general register storage operand Address. 


3) 


20 


OP 

OPX 

0 

LEH 

RE 

RR Forra 

1 1 i 

! 1 I 


■ f 

-I 1 f 



3 4 


7 8 


1 1 
] 2 



OP 

OPX 

1 

LEH 

RB 

3S 

: i .1 

s 1 1 


1 1 

. i i i 


3 4 


7 8 


1 1 
1 2 


RS Form 


In addition to an Operation Code field, the 3PG1 format coiuama: 


30 


Bit* 4-7 
BR 8 

Bit* $-11 


Bits 12-1* 


SPC Format 2 


OP Extension - ihe OPX field is used to extend the operation code. 

Zero or One - This bit being zero defines the operation io be a register operation. This 

btf being one defines tie operation to be a register storage operation. 

Operation Length - These bits are considered an unsigned value which la used to 

specify the leagft of Ins operand in 18 bit words. A value of zero corresponds to a length 

of one, and a value ct B'lir corresponds to a length of at&hL 

Register Address a - lb& RB field is used to specify vrfiich of the 10 oenerel registers is 

to be used as a storage address for fro operand. 


ihe Spedal (SPC2) format provides one genera! register storage operand address. 
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OP 

-J. I l_ 

RA 

I 1 1 

OPX 
ill 

R8 

i i i 

& 3 

4 7 

8 XI 1 


1 2 


to audition to an Operation Cods field, the SPC2 formal contains: 

BBS 4-7 Register Atwrese 1 - The RA IfeW is used to specify which of the IB general regretere is 
10 te u«d 9$ i*n operand end A* destination. 
75 Bfta OP Exferttion - The OPX field is used to extend ihe operation code, 

Bits 12-15 Register Address 2 - The RB Held is used to specify which of the 16 general registers is 
10 be used as a storage address for the operand. 

THE INSTRUCTION LIST OF THE ISA INCLUDES THE FOLLOWING: 
Table 1 (Page 1 of 3>. Fked- Point Angelic toti ructions 

Ci&ME » TYPME 

ada OA 


30 


33 


4* 


90 


65 
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Table 1 (Pag* U Crt 3>. Fcred-Poiftl Arilf»ro*fo to$trus5or« 

NAM! MM& TYPME 

MOMIC 

9 




3 



WITH DELTAS 

C4VWJ 

rvo 


Ann iiwiMPniATF 


X\M 


/»Af|TM r\CT TA'i 

^WIIH L/cLIA) 


Kl 




KK 


^AkllOA DC rXPD CPT A nPNDrcO 

_ _J _ 

DA 

J3 

rnf/IOADC 1 Kfl KjflCPfcl ATE 

UUlVirAKc llvlIvlcL/IAlc 

CI 

Rl 


(WITH DELTA; 

civrel 

Rl 


COMPARE FROM S lORAGE 

c 

RS 

so 

(WITH DELTA; 

cwd 

RS 


COMPARE REGISTER 

cr 

RR 


COPY 

cpy 

RS 


{WITH DELTA) 

cpywd 

RS 


LOrY WhH HOJH IMWcDIATc 

k * 

cpybi 

Rl 


{WITH DELTA) 

cpyhiwd 

Rl 

30 



Rl 

(WITH DELTA) 

cpylwd 

Rl 


^Vr« UMIXCvJ 

cpyda 

DA 


VslSr I U!UL.L> 1 1 IVj Iw I CW 1 f\ \ H 

cpydai 


38 


mc 

RS 


(WITH DELTA) 

Incwd 

RS 


LOAD DIRECT 

Ida 

DA 

40 

LOAD FROM STORAGE 

1 

RS 


(WITH DELTA) 

Iwd 

RS 


LOAD IMMEOIATE 

ii 

Rl 

4* 

(WITH DELTA) 

liwd 

Rl 


LOAD REGISTER 

Ir 

RR 


MULTIPLY SIGNED 

mpy 

SPC 

30 

MULTIPLY SIGNED EXTENDED 

mpyx 

SPC 


MULTIPLY SIGNED EXTENDED IMMEDIATE 

rnpyxi 

SPC 


56 


51 


EP 0 570 729 A2 


Table 1 <Pjrge 3 ot 3|, fixed- Pcioi Artfhnwn»c frTS$r<rctions 

NAME MNE- TYPI/IE 

s MOMIC 

MULTIPLY SIGNED IMMEDIATE mpyi SPC 

MULTIPLY UNSIGNED mpyu SPC 

MULTIPLY UNSIGNED EXTENDED tnpyux SPC 

MULTIPLY UNSIGNED EXTENDED IMMEDIATE mpyuxi SPC 

MULTIPLY UNSIGNED IMMEDIATE mpyut SPC 

STORE DIRECT Stda DA 

STORE st RS 

(WITH DELTA) stwd RS 

STORE IMMEDIATE sti RS 

v> (WITH DELTA} stiwd Ri 

SUBTRACT DIRECT sda DA 

SUBTRACT FROM STORAGE 5 RS 

{WJTH DELTA) swd RS 

SUBTRACT IMMEDIATE at RI 

WITH DELTA} aiwd R( 

SUBTRACT REGISTER sr RR 

SWAP AND EXCLUSIVE OR WITH STORAGE swapx RR 

X Table I (Page J 0 I 3). Siorage lo Storage Inslnjcliora 

NAME MWE» TYPME 

ADD STORAGE TO STORAGE sa SS 

(WITH DELTA) sawd SS 

ADD STORAGE TO STORAGE DIRECT aada SS 

ADD STORAGE TO STORAGE FINAL saf SS 

(WITH DELTA) safwd SS 

ADD STORAGE TO STORAGE FINAL DIRECT safda SS 

ADD STORAGE TO STORAGE INTERMEDIATE sal SS 

M (WITH DELTA) satwd SS 
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Table 2 {Page 3 Of 3). Stored to Stooge heiritttianc 

NAME Mlffii HEME 

MONIC 

ADO STORAGE TO STORAGE INTERMEDIATE 




S3 Ida 

oo 


ADD STORAGE TO STORAGE LOGICAL 

sal 

SS 

16 

(WITH DELTA) 

salwd 

SS 


ADD STORAGE TO STORAGE LOGICAL DIRECT 

salda 

ss 


COMPARE STORAGE TO STORAGE 

sc 

SS 

IS 

(WITH DELTA) 

$cwd 

SS 


COMPARE STORAGE TO STORAGE DIRECT 

scda 

SS 


COMPARE STORAGE TO STORAGE PINAL 

scf 

SS 

20 

(WITH DELTA) 

scfwd 

SS 


COMPARE STORAGE TO STORAGE FINAL DIRECT 

scfda 

SS 


COMPARE STORAGE TO STORAGE INTERMEDIATE 


SS 

2* 

(WITH DELTA) 

sciwd 

SS 


COMPARE STORAGE TO STORAGE INTERMEDIATE 




DIRECT 

scidd 

SS 


COMPARE STORAGE TO STORAGE LOGICAL 

scl 

SS 

30 

(WITH DELTA; 

sciwd 

SS 


COMPARE STORAGE TO SiORAGE LOGICAL 




DIRECT 

sdda 

SS 

3S 

MOVE STORAGE TO STORAGE 

smov 

ss 


(WITH DELTA) 

smovwd 

SS 


MOVE STORAGE TO STORAGE DIRECT 

srnovda 

SS 


SUBTRACT STORAGE TO STORAGE 
• 

ss 

SS 


(WITH DELTA) 

sswd 

SS 


SUBTRACT STORAGE TO STORAGE DIRECT 

ssda 

SS 


SUBTRACT STORAGE TO STORAGE FINAL 


SS 


(WITH DELTA) 

ssfwd 

ss 


SUBTRACT STORAGE TO STORAGE FINAL DIRECT 

ssfda 

ss 

50 

SUBTRACT STORAGE TO STORAGE INTERMEDIATE 

SSI 

SS 

(WITH DELTA} 

ggiwd 

ss 
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TMo 1 {Page 3 of J). Storage »c Storage Inductions 
NAME 

SUBTRACT STORAGE TO STORAGE INTERMEDIATE 
DIRECT 

SUBTRACT STORAGE TO STORAGE LOGICAL 

(WITH DELTA) 
SUBTRACT STORAGE TO STORAGE LOGICAL 

DIRECT 


MNE- 
M O NIC 

saida 
ssl 

sslwd 
sslda 


TVPME 

SS 

ss 

SS 

ss 


Table 3 


so 


Logical Instructlofta 

NAME 

MNEMONIC 

TVPME 

AND DIRECT ADDRESS 

nda 

DA 

AND FROM STORAGE 

n 

R8 

(WITH DELTA) 

nwd 

RS 

AND IMMEDIATE 

nl 

Rl 

(WITH DELTA) 

niwd 

Rl 

AND REGISTER 

nr 

RR 

OR DIRECT ADDRESS 


DA 

OR FROM STORAGE 

0 

RS 

(WITH DELTA) 


RS 

OR IMMEDIATE 

d 

Rl 

(WITH DELTA) 


Rl 

OR REGISTER 

or 

RR 

XOR DIRECT AD DRESS 

xcJa 

DA 

XOR FROM STORAGE 

X 

RS 

(WITH DELTA) 


RS 

XOR IMMEDIATE 

X! 

Rl 

(WITH DELTA) 


Rl 

XOR REGISTER 

yt 

RR 
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Table 4 |Pae<3 1 if T h Shin Instruction* 




NAME 


TYPft 



MON1C 


9 

vwnLC OllNAftY 

scale 

SPC 


oLHLfc dlNARY IMMEDIATE 

scafel 

SPC 


otALt BINARY REGIS! ER 

scaler 

SPC 

10 

bCALE HEXADECIMAL 

scaleh 

SPC 


SCALE HEXADECIMAL IMMEDIATE 

sea left i 

SPC 

ra 

SCALE HEXADECIMAL REGISTER 

scalehr 

SPC 


SHIFT LEFT ARITHMETIC BINARY 

sla 

SPC 


SHIFT LEFT ARITHMETIC BINARY IMMEDIATE 

slai 

SPC 


SHIFT LEFT ARITHMETIC BINARY REGISTER 

slar 

SPC 

SHIFT LEFT ARITHMETIC HEXADECIMAL 

slah 

SPC 


SHIFT U=FT ARITHMETIC HEXADECIMAL IMMEDIATE 

$Iahi 

SPC 

£3 

SHIFT LEFT ARITHMETIC HEXADECIMAL REGISTER 

slahr 

SPC 


SHIFT LEFT LOGJCAL BINARY 


SPC 


SHIFT LEFT LOGICAL BINARY IMMEDIATE 

Sill 

SPC 

3D 

SHIFT LEFT LOGICAL BINARY REGISTER 

slJr 

SPC 


SHIFT LEFT LOGICAL HEXADECIMAL 

sllh 

SPC 


SHIFT LEFT LOGICAL HEXADECIMAL IMMEDIATE 

sllhi 

SPC 

35 

SHIFT LEFT LOGICAL HEXADECIMAL REGISTER 

silhr 

SPC 


SHIFT RIGHT ARITHMETIC BINARY 

«ra 

SPC 


tiHlhr RIGHT ARITHMETIC BINARY IMMEDIATE 

srai 

SPC 


cniri rxioMJ ArClinMETiC BINARY REGISTER 

srar 

SPC 


SHIFT RIGHT ARITHMETIC HEXADECIMAL 

srah 

SPC 


SHIFT RIGHT ARITHMETIC HEXADECIMAL IMME- 

srahi 

SPC 


DIATE 




SHIFT RIGHT ARITHMETIC HEXADECIMAL REGISTER 

sreihr 

SPC 


SHIFT RIGHT LOGICAL 8INARY 


SPC 

30 

SHIFT RIGHT LOGICAL BINARY IMMEDIATE 

srli 

SPC 
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Tubla 4 <Paye ? of 2). Shift Instruction* 




NAME 

MNE- 

TYPME 





s 

onlrl Klvjrtl LUulLr\L DllNrtnCY rtCUIoi tin. 

srlf 

cop 


SHIFT RIGHT LOGICAL HEXADECIMAL 

srln 

SPC 


SHIFT RIGHT LOGICAL HEXADECIMAL IMMEDIATE 

srlhi 

SPC 

70 

SHIFT RIGHT LOGICAL HEXADECIMAL REGISTER 

srlhr 

SPC 

*5 

Tabic 5 (Page 1 of 2}. Brartta Instructions 




N AMP 

•vi c« 




IVIWIVI W 


20 


K 

r\0 




DQ 


BRANCH DIRECT 

bda 


29 



PI 



fc>iwcf 

Pi 


BRANCH REGISTER 

br 

RS 

30 

BRANCH AND LINK 



SRAMCH AND LINK DIRECT 

UcllU-a 

AA 
wr\ 


BRANCH AND LINK IMMEDIATE 

baii 

Rl 


(WITH DELTA* 

baliwd 

Rl 


BRANCH AND LINK REGISTER 

\ Villi Wl t tit \ +0 J » 1 v ■ % L^^aS I w ■ l_l % 

balr 

RS 


BRANCH BACKWARD 

bb 

RS 


tWIln UtLIM| 

ODWO 



BRANCH BACKWARD DIRECT 

bbda 

DA 


BRANCH BACKWARD IMMEDIATE 

bbl 

Rl 


(WITH DELTA) 

bblwd 

Rl 


BRANCH BACKWARD REGISTER 

bbr 

RS 


BRANCH FORWARD 

bf 

RS 


(WITH DELTA) 

bfwd 

RS 

SO 

BRANCH FORWARD DIRECT 

bfda 

DA 
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JO 


13 


Table 5 (frage 2 of 2}. Branch InotrtictiQns 
NAME 

BRANCH FORWARD IMMEDIATE 

(WITH DELTA) 
BRANCH FORWARD REGISTER 
BRANCH ON CONDITION 

(WITH DELTA) 
BRANCH ON CONDITION DIRECT 
BRANCH ON CONDITION IMMEDIATE 

(WITH DELTA) 
BRANCH ON CONDITION REGISTER 
BRANCH RELATIVE 

(WITH DELTA) 
NULL OPMERATIQN 


MONIC 
bfi 

bfiwd 

bfr 

be 

bewd 
beda 
bd 

bdvwd 

bcr 

brel 

brehvd 

nooj> 


TYPME 

Rl 

Rl 

RS 

RS 

RS 

RS 

Rl 

Rl 

RS 

Rl 

RS 

RR 


Status Switching Instiucflcna 

NAME 

MNEMONIC 


RETURN 

ret 

SPC j 


0$ 


Table 7 


40 


InpuKtotput Instructions 

NAME 

MNEMONIC 

TYPME 

IN 

IN 

SPC 

OUT 

OUT 

SPC 

INTERNAL DIOFVDIQW 

INTR 

SPC 


55 


SOME SUMMARY FEATURES 

The APAP Mgchlne to Perspective 

Wo have described m accordance with our invention cookJ be thought of In its more detaBed aspects to 
be positioned In the technology somewhere between the CM-i and N-oibe. Like out APAP I the CM-1 uses 
a point design for the processing element and combines processing elements wflh memory on the basic 
chip. The however uses a 1 bit wide serial processor .write ihe APAP series wttl use a 16 bit wws 
processor. The CM sejfea d* machines Varied wrfh 4K bif3 of memory per processor and hea qiow« to a or 
18K brt£ versus tfw 32K by 19 brcs we have provided ferine first APAP ohrp. The CM-1 and to fcttarons 
ere strictly SlMD machine* while lie CM-5 Is a hybrid. Instead of this, our APAP will effectively use MIMD 
operating modes in conjunction wfth SIMD modes when useM While our parallel 16 IM wide PMEs might 
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be vterved as a step toward me N-cubs, mis step is not warranted. Tha APAP does not separate memory 
and rouyng from the processing element as doas the N-cube kind of machine. AJeo, the APAP provide* for 
if) to 32K 16 bit PMEs while me N-cube Only provides for 4K 32 bit processors. 

Even with ihe superficial similarities pneeenied above, the APAP concept comply differs from the 
s CM and M-cube series by: 

1. The motived hypercube incorporated in ow APAP is a new invention providing a significant packaging 
and addressing advantage when compared with hypercube topologies. For instance, consider that the 
32K PME APAP In He first preferred embodiment has a network diameter of 19 logical steps end. with 
transparency, this can be reduced % en effective 16 logical step*. Further, by cornpsrison. n a pure 

jo hypercube v«re uced, end if all PMEs were sending data through an 8 step pati, then on average 2 of 
every 8 PMEs would be active white the remainder would be delayed due to blockage. 

Alternatively, consider the G4K hypercube that would be needed it CM-1 was a pore hypercube. In 
that casa, each PME would require ports to 16 other PMEs, and data could be routed between the two 
farthest separated PMEs tn i& logical steps, ff ell PMEs tried to transfer art average distance of 7 steps, 

w the 2 of every 7 would be ecfivo, However, CM-1 doss not utilize e 16d hypercube. & interconnects the 
16 nodes on a chip with a NEWS network; then it provides one rouier function wtthfci the chip. The 4096 
routers are connected into a 1 2d hypercube Wan no collisions ihe hybrid sttf has a lop>cel diameter of 
15, but since id PMEs could be contending for the tin* H& effective diameter is much greater. That in, 
wUh 8 step moves only 2 of 16 PMEs could be active, which means that 8 complete cycles rather than 4 

so cycles ere needed to complete eB data moves. 

The N-cube actually uaSzee a pure hypercube, but currently only provides for a 4696 PMEs and 
thus, utilizes a i2d {1 3d (or 8192 PMEs) hypercube. For the N-cutoe to grow to i6K processors, at vdiich 
point it would have me same processing data width as the APAP, it would have to add tour times as 
much hardware and would have to increase the connection porfe to each PME router by 25%, Although 

25 no hard date exists to support this conclusion, would appear that the N-cube architecture runs cut of 
connector pin* prior io reaching a i6K PME maewne. 

2. The completely integrated and distributed nature ot major lasks within the APAP machine is a decided 
advantage. As was noted for the CM and N-cube aeries of machines, each had to have separate units for 
message routing as weM as separate units for floating point coprocessors, The APAP system combines 

30 the integer, Heating point processing* message rooting and I/O control into flie single point design PME. 
That design is then replicated 8 iimes on a chip, and the chip is fen replicated 4K times to produce the 
array* This provides severe! advances; 

a. Using one chip means maximum size production rune and minima] system factor coei& 

b. R&guter architecture pf oduces the most effective programming systems. 

as c. Almost an chip pine can be docScaied to to generic problem of irrterprocosaor communication, 

maximizing the Inter-chip I/O bandwidth which fends to be a important fonHing factor in MPP designs. 

3. The APAP has the unique design abHHy to take advantage oi chip techndogy gains end capital 
investment in custom chip designs, 

Consider me question of floating point performance It Is anticipated that APAP PME performance on 
<tt DAKPY will be about 125 cycles per flop. In contrast, fte 1 387 Coprocessor would be about 14 cycles 
while the Weiteo Coprocessor in (he CM-1 would be about 6 cycles. However, in fa CM case th&ro is 
only one floating point unlfc for every 10 PMEs while m the N-cube case there is probably one type 
chip associated wlih each ot the *3£$ processors. Our APAP has 16 times as many PMEs and therefore 
can almost completely make up tor the single unit performance delta. 
45 Mors significantly, me 8 apap pmes within a chip are constructed from 50K gates currently 

available in the technology. As memory macros shrink and the number ot gsies avail&bte lo the logic 
mcreasss. Spending thai increase on enhanced floating point normaftzaUon should permit APAP Heating 
point performance to tar exceed me other unite. Alternatively, effort could be spent to generate a PME ci 
PME subsection design using custom design approaches, enhancing totel performance while in no way 
so affecting any S/W developed for the machine. 

We believe our design for our APAP has characteristics poised to take aefcaniage oi tl* futum 
process technology growth. In contrast, ins nearest similar machines CM-x and N-cube which employ a 
system like that described In FIGURE 1 seem well poised to take advantage of yesterday's technology 
which m feet Is dead ended. 
S3 An advantage of tho APAP concept is the zttfitf lo use OA 3 D associated with groups of PMEs. This 
apap capability, as *ei as the abiiHy to connect displays and auxiliary storage, is a by-product of picking 
MC bus structures as me interface to the external CO ports of the PME Array, Thus, APAP systems will be 
configurable and can include card mounted hard drives selected {rem one of me set of unite that are 
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compatible with PStf or RISCflBOOO units. Furiher. that capability shou&d be available without designing any 
additional part number modules altfiough it does require utilizing more replications of the bacfcpanel and 
base enclosure then does the APAR 

This brief perspective © no4 intended to be Iwrrierig, but rather is intended to cause those skied in the 

a art to review- the foregoing description and examine how the many Inventions we have described which may 
be used to rnova End ar; of massively parallel systems ahead to a Bme when programing to no longer a 
significant problem and the costs of such systems are much lower. Our tend of system can be made 
available, not only to the few, but to many as it could be made at a coat vvithkn the reach of commercial 
department level procurement*. 

70 While vre have described our preferred embodiments of our invention, it will be understood that those 
skilled in the art both now and in the future, upon the understanding of these discussions will make various 
Improvements and enhancements thereto which fall within the scope of the claims which follow. These 
dams should be construed to maintain the proper protection for She invention first disclosed. 

;s Claim* 

1. A computer system comprising, a plurality of muffl-precessor memory elements, each having commu- 
nication paths, processor and memory, and wherein a programmable router is provided for routing data 
and control Information from one multiprocessor memory element to another multi-processor memory 

ao element and between nodes of the computer system. 

2. A computer system according to claim i wherein each mura-croceasor memory element <PME) has 2n 
processors, and communication paths which minimize delays due to chip crossings, 

23 & A computer system according to claim 1 wherein each murfr processor memory element cPME} ha* a 
processor, memory and rcoiers win in a single chip and internal and external coitwrurticaiion paths 
which minimize delays due to chip crossings, each processor memory element having means tor fixed 
and floating point processing, rousing and I/O contreL 

30 4. A computer system according to claim t further comprising within a processor memory element 

a nefive instruction set means for providing an expandable multiply function, a programme* router for 
routing information dtsmstively tefttighi, 

NEWS matrix, NEWS/up-down. hypercufce. and wherein said programmable router ia empfcys a 
hardwired distributed router provided by each processing memory element. 

as 

5. A computer computer system according to claim 1 organized as a massively parallel machine *vitf> 
nodes interconnected as a n dimenebnai network cluster with parallel communication para between 
processor memory elements along sakJ internal and externa! communication paihs providing a process- 
ing arrey> and wherein processing memory elements of an ewey have a tiensperent mode ut»zed v*ien 

<o routing data betoeen processing memory elements within a chip sat ot processing memory elements 
for permitting reduction of the effeciwe network diameter of a network of nodes, 

6. A computer system according to claim i wherein a node of a processor array has mUHpie stogie 
processor element made up of 32K 16-bit ttorxfo with a 16-bil processor for a network node of eight 

« processors with the* associated memory wan their My distributed I/O routers end signal I/O ports. 

7. A computer system according Ic claim 1 wlweln a node of a processor array has muJUpld single 
processor elements made up of 32K ie-b*t wards wfeh a processor for a netwoik node of eight 
processors with their associated memory mfa their fully disputed \rQ routers and signal VO ports 

so combined as groups groups of node clusters organized as a 2d modified hypercube, 

& A computer sysrern according ic claim i wherein a node oi a processor array has muhfpie single 
processor elements made up of 32K 16-bit wrxto w*h a 10-bH processor tor a network node of eight 
processors with tnetr associated memory wffri their fulry distributed VO routers and signal K> pons 
58 combined as groups groups of node clusters organized as a 2d modffied hypercube, with up lo 04 
dusters integrated In a network of node clusters to form are integrated to form a id mooWed 
hypercube of up to 32,760 processing memory elements. 
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9. A computer system according to claim 1 whereto a node processing memory atarnerri has internal data 
flouts using high epeed hard registers to feed distributed ALU and and LO router registers a/kd toc^o for 
$1 opeidtiGrt$, 

10l A computer system according to datm 1 wherein a node processing memory element In has an VO 
port for of* cfep byte wide oommunic&tion, and has input pom that are connected *ucn the* date may 
be routed from input to memory, or from an input address register to an output register via a direct 
parallel data path. 

it* A compute! system according to claim 1 wherein a node has multiple processor memory elements and 
is connected to other nodes to a duster network with data routing distributed between hardware and 
software, wth soflvtfre controlling mo$* o* the task sequencing function, 

12. A computer system according to claim 1 wherein a node has multiple processor memory elements and 
is connected to other nodes in a cluster network wfih date routing distributed between hardware and 
software, tttih hardware provided lor performing Inner loop transfers and minimizing overhead on the 
outer loops of the node. 

13. A computer system according to claim 1 wherein a node has multiple processor memory etemems end 
is competed to other node* in a duster network wif> I/O programs at dedicated interrupt levels «oi 
managing the network. 

14. A computer «.y*i*ro according to claim 1 •vherein a node has multiple processor memory element* and 
is connected to other nodes in a cluster network with I/O programs at dedicated interrupt levels for 
managing the network* each processor memory element havlne interrupt registers end dedicating tour 
interrupt levers io receiving dale from four neighbors, a buHer provided at each level by loading 
registers at the level, and having m and return Instruction pairs using a buffer address and transfer 
count to enable the processor memory element to accept wards from an input bus and to s»re mem to 
toe bwter, 

-ts. A multi-processor memory system, comprising: a ptuneJrty of multi-processor memory elements, each 
muS'proceeeor memory etemsnt (PME) having 2n processors, memory and routers within a Erngle chip 
and internal and externa! comrnunlcatlon pethe which mlnimrte delays due to chip crossings. 
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